[==========] Running 6 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from RaftConfigChangeITest
[ RUN ] RaftConfigChangeITest.TestKudu2147
2026-11-19T09:03:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-11-19T09:03:58Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20261119 09:03:58.786558 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.252.126:33525
--webserver_interface=127.30.252.126
--webserver_port=0
--builtin_ntp_servers=127.30.252.84:37035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.252.126:33525 with env {}
W20261119 09:03:58.889669 31737 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:03:58.889864 31737 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:03:58.889920 31737 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:03:58.893496 31737 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261119 09:03:58.893567 31737 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:03:58.893603 31737 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261119 09:03:58.893630 31737 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261119 09:03:58.897850 31737 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:37035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.252.126:33525
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.252.126:33525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.252.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.31737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:03:58.898665 31737 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:03:58.899385 31737 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:03:58.905232 31742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:03:58.905224 31745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:03:58.905295 31737 server_base.cc:1061] running on GCE node
W20261119 09:03:58.905231 31743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:03:58.905810 31737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:03:58.906524 31737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:03:58.907655 31737 hybrid_clock.cc:648] HybridClock initialized: now 1795079038907630 us; error 40 us; skew 500 ppm
I20261119 09:03:58.909341 31737 webserver.cc:577] Webserver started at http://127.30.252.126:44005/ using document root <none> and password file <none>
I20261119 09:03:58.909915 31737 fs_manager.cc:362] Metadata directory not provided
I20261119 09:03:58.909969 31737 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:03:58.910183 31737 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:03:58.911617 31737 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/master-0/data/instance:
uuid: "78e15c55d0a84559b465f08040361180"
format_stamp: "Formatted at 2026-11-19 09:03:58 on dist-test-slave-1fcg"
I20261119 09:03:58.912050 31737 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/master-0/wal/instance:
uuid: "78e15c55d0a84559b465f08040361180"
format_stamp: "Formatted at 2026-11-19 09:03:58 on dist-test-slave-1fcg"
I20261119 09:03:58.915366 31737 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20261119 09:03:58.917742 31751 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:03:58.918823 31737 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261119 09:03:58.918946 31737 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/master-0/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/master-0/wal
uuid: "78e15c55d0a84559b465f08040361180"
format_stamp: "Formatted at 2026-11-19 09:03:58 on dist-test-slave-1fcg"
I20261119 09:03:58.919039 31737 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:03:58.933130 31737 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:03:58.933720 31737 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:03:58.933871 31737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:03:58.941205 31737 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.126:33525
I20261119 09:03:58.941226 31803 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.126:33525 every 8 connection(s)
I20261119 09:03:58.942483 31737 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/master-0/data/info.pb
I20261119 09:03:58.945483 31804 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:03:58.950343 31804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180: Bootstrap starting.
I20261119 09:03:58.952175 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 31737
I20261119 09:03:58.952307 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/master-0/wal/instance
I20261119 09:03:58.952809 31804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180: Neither blocks nor log segments found. Creating new log.
I20261119 09:03:58.953538 31804 log.cc:826] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180: Log is configured to *not* fsync() on all Append() calls
I20261119 09:03:58.955749 31804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180: No bootstrap required, opened a new log
I20261119 09:03:58.959270 31804 raft_consensus.cc:360] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e15c55d0a84559b465f08040361180" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 33525 } }
I20261119 09:03:58.959487 31804 raft_consensus.cc:386] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:03:58.959546 31804 raft_consensus.cc:748] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e15c55d0a84559b465f08040361180, State: Initialized, Role: FOLLOWER
I20261119 09:03:58.959934 31804 consensus_queue.cc:261] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e15c55d0a84559b465f08040361180" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 33525 } }
I20261119 09:03:58.960060 31804 raft_consensus.cc:400] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261119 09:03:58.960134 31804 raft_consensus.cc:499] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261119 09:03:58.960223 31804 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180 [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:03:58.961175 31804 raft_consensus.cc:521] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e15c55d0a84559b465f08040361180" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 33525 } }
I20261119 09:03:58.961537 31804 leader_election.cc:304] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78e15c55d0a84559b465f08040361180; no voters:
I20261119 09:03:58.961809 31804 leader_election.cc:290] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261119 09:03:58.961891 31809 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180 [term 1 FOLLOWER]: Leader election won for term 1
I20261119 09:03:58.962066 31809 raft_consensus.cc:705] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180 [term 1 LEADER]: Becoming Leader. State: Replica: 78e15c55d0a84559b465f08040361180, State: Running, Role: LEADER
I20261119 09:03:58.962430 31809 consensus_queue.cc:238] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e15c55d0a84559b465f08040361180" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 33525 } }
I20261119 09:03:58.963240 31804 sys_catalog.cc:565] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180 [sys.catalog]: configured and running, proceeding with master startup.
I20261119 09:03:58.963939 31811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78e15c55d0a84559b465f08040361180. Latest consensus state: current_term: 1 leader_uuid: "78e15c55d0a84559b465f08040361180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e15c55d0a84559b465f08040361180" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 33525 } } }
I20261119 09:03:58.964164 31811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180 [sys.catalog]: This master's current role is: LEADER
I20261119 09:03:58.964382 31810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78e15c55d0a84559b465f08040361180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78e15c55d0a84559b465f08040361180" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 33525 } } }
I20261119 09:03:58.964524 31810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180 [sys.catalog]: This master's current role is: LEADER
I20261119 09:03:58.964661 31817 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261119 09:03:58.968166 31817 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261119 09:03:58.974319 31817 catalog_manager.cc:1398] Generated new cluster ID: c48ce6159a174183be9eb816d6525ed6
I20261119 09:03:58.974399 31817 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261119 09:03:58.993113 31817 catalog_manager.cc:1421] Generated new certificate authority record
I20261119 09:03:58.994153 31817 catalog_manager.cc:1555] Loading token signing keys...
I20261119 09:03:59.001073 31817 catalog_manager.cc:6237] T 00000000000000000000000000000000 P 78e15c55d0a84559b465f08040361180: Generated new TSK 0
I20261119 09:03:59.002141 31817 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261119 09:03:59.005705 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.65:0
--local_ip_for_outbound_sockets=127.30.252.65
--webserver_interface=127.30.252.65
--webserver_port=0
--tserver_master_addrs=127.30.252.126:33525
--builtin_ntp_servers=127.30.252.84:37035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=3000 with env {}
W20261119 09:03:59.116084 31828 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:03:59.116279 31828 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:03:59.116353 31828 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:03:59.120527 31828 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:03:59.120772 31828 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.65
I20261119 09:03:59.125375 31828 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:37035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=3000
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.252.65
--webserver_port=0
--tserver_master_addrs=127.30.252.126:33525
--never_fsync=true
--heap_profile_path=/tmp/kudu.31828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:03:59.126164 31828 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:03:59.126842 31828 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:03:59.132731 31833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:03:59.132812 31834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:03:59.132745 31836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:03:59.133199 31828 server_base.cc:1061] running on GCE node
I20261119 09:03:59.133598 31828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:03:59.134199 31828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:03:59.135357 31828 hybrid_clock.cc:648] HybridClock initialized: now 1795079039135330 us; error 37 us; skew 500 ppm
I20261119 09:03:59.137537 31828 webserver.cc:577] Webserver started at http://127.30.252.65:34387/ using document root <none> and password file <none>
I20261119 09:03:59.138216 31828 fs_manager.cc:362] Metadata directory not provided
I20261119 09:03:59.138270 31828 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:03:59.138496 31828 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:03:59.140157 31828 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-0/data/instance:
uuid: "ad7b4b42523b489f8ad7142d774cf5b7"
format_stamp: "Formatted at 2026-11-19 09:03:59 on dist-test-slave-1fcg"
I20261119 09:03:59.140717 31828 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-0/wal/instance:
uuid: "ad7b4b42523b489f8ad7142d774cf5b7"
format_stamp: "Formatted at 2026-11-19 09:03:59 on dist-test-slave-1fcg"
I20261119 09:03:59.144794 31828 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20261119 09:03:59.147298 31842 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:03:59.148406 31828 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261119 09:03:59.148504 31828 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-0/wal
uuid: "ad7b4b42523b489f8ad7142d774cf5b7"
format_stamp: "Formatted at 2026-11-19 09:03:59 on dist-test-slave-1fcg"
I20261119 09:03:59.148582 31828 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:03:59.163705 31828 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:03:59.164188 31828 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:03:59.164314 31828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:03:59.164911 31828 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:03:59.165992 31828 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:03:59.166040 31828 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:03:59.166085 31828 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:03:59.166098 31828 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:03:59.174345 31828 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.65:45147
I20261119 09:03:59.174376 31955 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.65:45147 every 8 connection(s)
I20261119 09:03:59.175338 31828 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-0/data/info.pb
I20261119 09:03:59.182009 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 31828
I20261119 09:03:59.182169 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-0/wal/instance
I20261119 09:03:59.183548 31956 heartbeater.cc:345] Connected to a master server at 127.30.252.126:33525
I20261119 09:03:59.183830 31956 heartbeater.cc:462] Registering TS with master...
I20261119 09:03:59.184383 31956 heartbeater.cc:508] Master 127.30.252.126:33525 requested a full tablet report, sending...
I20261119 09:03:59.184589 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.66:0
--local_ip_for_outbound_sockets=127.30.252.66
--webserver_interface=127.30.252.66
--webserver_port=0
--tserver_master_addrs=127.30.252.126:33525
--builtin_ntp_servers=127.30.252.84:37035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=3000 with env {}
I20261119 09:03:59.186121 31768 ts_manager.cc:210] Registered new tserver with Master: ad7b4b42523b489f8ad7142d774cf5b7 (127.30.252.65:45147)
I20261119 09:03:59.187673 31768 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.65:52839
W20261119 09:03:59.287880 31959 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:03:59.288043 31959 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:03:59.288084 31959 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:03:59.291693 31959 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:03:59.291810 31959 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.66
I20261119 09:03:59.295943 31959 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:37035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=3000
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.252.66
--webserver_port=0
--tserver_master_addrs=127.30.252.126:33525
--never_fsync=true
--heap_profile_path=/tmp/kudu.31959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:03:59.296722 31959 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:03:59.297420 31959 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:03:59.303870 31964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:03:59.303891 31965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:03:59.304100 31967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:03:59.304430 31959 server_base.cc:1061] running on GCE node
I20261119 09:03:59.304798 31959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:03:59.305364 31959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:03:59.306535 31959 hybrid_clock.cc:648] HybridClock initialized: now 1795079039306513 us; error 34 us; skew 500 ppm
I20261119 09:03:59.308194 31959 webserver.cc:577] Webserver started at http://127.30.252.66:45235/ using document root <none> and password file <none>
I20261119 09:03:59.308759 31959 fs_manager.cc:362] Metadata directory not provided
I20261119 09:03:59.308809 31959 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:03:59.309010 31959 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:03:59.310328 31959 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-1/data/instance:
uuid: "5a6c5a6e8b664dfa846f479be75116e4"
format_stamp: "Formatted at 2026-11-19 09:03:59 on dist-test-slave-1fcg"
I20261119 09:03:59.310746 31959 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-1/wal/instance:
uuid: "5a6c5a6e8b664dfa846f479be75116e4"
format_stamp: "Formatted at 2026-11-19 09:03:59 on dist-test-slave-1fcg"
I20261119 09:03:59.313961 31959 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261119 09:03:59.316578 31973 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:03:59.317767 31959 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261119 09:03:59.317893 31959 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-1/wal
uuid: "5a6c5a6e8b664dfa846f479be75116e4"
format_stamp: "Formatted at 2026-11-19 09:03:59 on dist-test-slave-1fcg"
I20261119 09:03:59.317984 31959 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:03:59.363215 31959 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:03:59.363811 31959 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:03:59.363958 31959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:03:59.364517 31959 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:03:59.365480 31959 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:03:59.365527 31959 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:03:59.365583 31959 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:03:59.365612 31959 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:03:59.374099 31959 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.66:46023
I20261119 09:03:59.374152 32086 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.66:46023 every 8 connection(s)
I20261119 09:03:59.375124 31959 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-1/data/info.pb
I20261119 09:03:59.381003 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 31959
I20261119 09:03:59.381105 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-1/wal/instance
I20261119 09:03:59.383610 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.67:0
--local_ip_for_outbound_sockets=127.30.252.67
--webserver_interface=127.30.252.67
--webserver_port=0
--tserver_master_addrs=127.30.252.126:33525
--builtin_ntp_servers=127.30.252.84:37035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=3000 with env {}
I20261119 09:03:59.384003 32087 heartbeater.cc:345] Connected to a master server at 127.30.252.126:33525
I20261119 09:03:59.384227 32087 heartbeater.cc:462] Registering TS with master...
I20261119 09:03:59.384701 32087 heartbeater.cc:508] Master 127.30.252.126:33525 requested a full tablet report, sending...
I20261119 09:03:59.385766 31768 ts_manager.cc:210] Registered new tserver with Master: 5a6c5a6e8b664dfa846f479be75116e4 (127.30.252.66:46023)
I20261119 09:03:59.386339 31768 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.66:56755
W20261119 09:03:59.487880 32090 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:03:59.488066 32090 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:03:59.488124 32090 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:03:59.491681 32090 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:03:59.491801 32090 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.67
I20261119 09:03:59.495733 32090 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:37035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=3000
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.252.67
--webserver_port=0
--tserver_master_addrs=127.30.252.126:33525
--never_fsync=true
--heap_profile_path=/tmp/kudu.32090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:03:59.496475 32090 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:03:59.497152 32090 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:03:59.502990 32098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:03:59.503017 32095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:03:59.503043 32096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:03:59.503257 32090 server_base.cc:1061] running on GCE node
I20261119 09:03:59.503655 32090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:03:59.504194 32090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:03:59.505391 32090 hybrid_clock.cc:648] HybridClock initialized: now 1795079039505364 us; error 57 us; skew 500 ppm
I20261119 09:03:59.507048 32090 webserver.cc:577] Webserver started at http://127.30.252.67:34449/ using document root <none> and password file <none>
I20261119 09:03:59.507637 32090 fs_manager.cc:362] Metadata directory not provided
I20261119 09:03:59.507690 32090 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:03:59.507893 32090 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:03:59.509196 32090 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-2/data/instance:
uuid: "a82d2f5d912a42a7bea8465837b33149"
format_stamp: "Formatted at 2026-11-19 09:03:59 on dist-test-slave-1fcg"
I20261119 09:03:59.509634 32090 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-2/wal/instance:
uuid: "a82d2f5d912a42a7bea8465837b33149"
format_stamp: "Formatted at 2026-11-19 09:03:59 on dist-test-slave-1fcg"
I20261119 09:03:59.512795 32090 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20261119 09:03:59.515326 32104 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:03:59.516893 32090 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20261119 09:03:59.517093 32090 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-2/wal
uuid: "a82d2f5d912a42a7bea8465837b33149"
format_stamp: "Formatted at 2026-11-19 09:03:59 on dist-test-slave-1fcg"
I20261119 09:03:59.517217 32090 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:03:59.541451 32090 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:03:59.541978 32090 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:03:59.542120 32090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:03:59.542667 32090 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:03:59.543732 32090 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:03:59.543780 32090 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:03:59.543836 32090 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:03:59.543865 32090 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:03:59.552912 32090 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.67:44277
I20261119 09:03:59.552965 32217 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.67:44277 every 8 connection(s)
I20261119 09:03:59.553902 32090 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-2/data/info.pb
I20261119 09:03:59.559803 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 32090
I20261119 09:03:59.560029 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-2/wal/instance
I20261119 09:03:59.562919 32218 heartbeater.cc:345] Connected to a master server at 127.30.252.126:33525
I20261119 09:03:59.563165 32218 heartbeater.cc:462] Registering TS with master...
I20261119 09:03:59.563807 32218 heartbeater.cc:508] Master 127.30.252.126:33525 requested a full tablet report, sending...
I20261119 09:03:59.565163 31768 ts_manager.cc:210] Registered new tserver with Master: a82d2f5d912a42a7bea8465837b33149 (127.30.252.67:44277)
I20261119 09:03:59.565791 31768 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.67:54955
I20261119 09:03:59.576607 31729 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20261119 09:03:59.603159 31768 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55616:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20261119 09:03:59.604429 31768 catalog_manager.cc:7244] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261119 09:03:59.622924 32021 tablet_service.cc:1495] Processing CreateTablet for tablet bdebe591f2aa4d98b1a538f011b398f3 (DEFAULT_TABLE table=test-workload [id=53f572791d434587895f156ca794a324]), partition=RANGE (key) PARTITION UNBOUNDED
I20261119 09:03:59.624084 32021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdebe591f2aa4d98b1a538f011b398f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:03:59.625065 32152 tablet_service.cc:1495] Processing CreateTablet for tablet bdebe591f2aa4d98b1a538f011b398f3 (DEFAULT_TABLE table=test-workload [id=53f572791d434587895f156ca794a324]), partition=RANGE (key) PARTITION UNBOUNDED
I20261119 09:03:59.625029 31890 tablet_service.cc:1495] Processing CreateTablet for tablet bdebe591f2aa4d98b1a538f011b398f3 (DEFAULT_TABLE table=test-workload [id=53f572791d434587895f156ca794a324]), partition=RANGE (key) PARTITION UNBOUNDED
I20261119 09:03:59.626032 32152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdebe591f2aa4d98b1a538f011b398f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:03:59.626032 31890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdebe591f2aa4d98b1a538f011b398f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:03:59.629767 32240 tablet_bootstrap.cc:492] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4: Bootstrap starting.
I20261119 09:03:59.632484 32240 tablet_bootstrap.cc:654] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4: Neither blocks nor log segments found. Creating new log.
I20261119 09:03:59.632704 32242 tablet_bootstrap.cc:492] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149: Bootstrap starting.
I20261119 09:03:59.632880 32241 tablet_bootstrap.cc:492] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: Bootstrap starting.
I20261119 09:03:59.633654 32240 log.cc:826] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4: Log is configured to *not* fsync() on all Append() calls
I20261119 09:03:59.635245 32241 tablet_bootstrap.cc:654] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: Neither blocks nor log segments found. Creating new log.
I20261119 09:03:59.636070 32242 tablet_bootstrap.cc:654] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149: Neither blocks nor log segments found. Creating new log.
I20261119 09:03:59.636415 32240 tablet_bootstrap.cc:492] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4: No bootstrap required, opened a new log
I20261119 09:03:59.636722 32241 log.cc:826] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: Log is configured to *not* fsync() on all Append() calls
I20261119 09:03:59.636775 32240 ts_tablet_manager.cc:1449] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20261119 09:03:59.637279 32242 log.cc:826] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149: Log is configured to *not* fsync() on all Append() calls
I20261119 09:03:59.638595 32241 tablet_bootstrap.cc:492] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: No bootstrap required, opened a new log
I20261119 09:03:59.638849 32241 ts_tablet_manager.cc:1449] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20261119 09:03:59.639392 32242 tablet_bootstrap.cc:492] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149: No bootstrap required, opened a new log
I20261119 09:03:59.639705 32242 ts_tablet_manager.cc:1449] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20261119 09:03:59.641213 32240 raft_consensus.cc:360] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } } peers { permanent_uuid: "ad7b4b42523b489f8ad7142d774cf5b7" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 45147 } }
I20261119 09:03:59.641494 32240 raft_consensus.cc:386] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:03:59.641598 32240 raft_consensus.cc:748] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a6c5a6e8b664dfa846f479be75116e4, State: Initialized, Role: FOLLOWER
I20261119 09:03:59.641999 32241 raft_consensus.cc:360] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } } peers { permanent_uuid: "ad7b4b42523b489f8ad7142d774cf5b7" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 45147 } }
I20261119 09:03:59.642165 32241 raft_consensus.cc:386] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:03:59.642089 32240 consensus_queue.cc:261] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } } peers { permanent_uuid: "ad7b4b42523b489f8ad7142d774cf5b7" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 45147 } }
I20261119 09:03:59.642239 32241 raft_consensus.cc:748] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad7b4b42523b489f8ad7142d774cf5b7, State: Initialized, Role: FOLLOWER
I20261119 09:03:59.642895 32241 consensus_queue.cc:261] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } } peers { permanent_uuid: "ad7b4b42523b489f8ad7142d774cf5b7" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 45147 } }
I20261119 09:03:59.643265 32240 ts_tablet_manager.cc:1480] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20261119 09:03:59.643684 31956 heartbeater.cc:500] Master 127.30.252.126:33525 was elected leader, sending a full tablet report...
I20261119 09:03:59.643715 32087 heartbeater.cc:500] Master 127.30.252.126:33525 was elected leader, sending a full tablet report...
I20261119 09:03:59.643738 32241 ts_tablet_manager.cc:1480] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20261119 09:03:59.644109 32242 raft_consensus.cc:360] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } } peers { permanent_uuid: "ad7b4b42523b489f8ad7142d774cf5b7" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 45147 } }
I20261119 09:03:59.644376 32242 raft_consensus.cc:386] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:03:59.644466 32242 raft_consensus.cc:748] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a82d2f5d912a42a7bea8465837b33149, State: Initialized, Role: FOLLOWER
I20261119 09:03:59.644930 32242 consensus_queue.cc:261] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } } peers { permanent_uuid: "ad7b4b42523b489f8ad7142d774cf5b7" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 45147 } }
I20261119 09:03:59.645694 32218 heartbeater.cc:500] Master 127.30.252.126:33525 was elected leader, sending a full tablet report...
I20261119 09:03:59.645975 32242 ts_tablet_manager.cc:1480] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20261119 09:03:59.676409 31957 tablet.cc:2554] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20261119 09:03:59.804854 32219 tablet.cc:2554] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20261119 09:03:59.876281 32088 tablet.cc:2554] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261119 09:04:00.429090 32249 raft_consensus.cc:499] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261119 09:04:00.429288 32249 raft_consensus.cc:521] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } } peers { permanent_uuid: "ad7b4b42523b489f8ad7142d774cf5b7" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 45147 } }
I20261119 09:04:00.430480 32249 leader_election.cc:290] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a6c5a6e8b664dfa846f479be75116e4 (127.30.252.66:46023), ad7b4b42523b489f8ad7142d774cf5b7 (127.30.252.65:45147)
I20261119 09:04:00.435220 31910 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "bdebe591f2aa4d98b1a538f011b398f3" candidate_uuid: "a82d2f5d912a42a7bea8465837b33149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad7b4b42523b489f8ad7142d774cf5b7" is_pre_election: true
I20261119 09:04:00.435220 32041 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "bdebe591f2aa4d98b1a538f011b398f3" candidate_uuid: "a82d2f5d912a42a7bea8465837b33149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a6c5a6e8b664dfa846f479be75116e4" is_pre_election: true
I20261119 09:04:00.435526 32041 raft_consensus.cc:2490] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a82d2f5d912a42a7bea8465837b33149 in term 0.
I20261119 09:04:00.435526 31910 raft_consensus.cc:2490] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a82d2f5d912a42a7bea8465837b33149 in term 0.
I20261119 09:04:00.436347 32106 leader_election.cc:304] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a6c5a6e8b664dfa846f479be75116e4, a82d2f5d912a42a7bea8465837b33149; no voters:
I20261119 09:04:00.436623 32249 raft_consensus.cc:2826] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261119 09:04:00.436676 32249 raft_consensus.cc:499] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261119 09:04:00.436705 32249 raft_consensus.cc:3096] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:00.437633 32249 raft_consensus.cc:521] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } } peers { permanent_uuid: "ad7b4b42523b489f8ad7142d774cf5b7" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 45147 } }
I20261119 09:04:00.437987 32249 leader_election.cc:290] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [CANDIDATE]: Term 1 election: Requested vote from peers 5a6c5a6e8b664dfa846f479be75116e4 (127.30.252.66:46023), ad7b4b42523b489f8ad7142d774cf5b7 (127.30.252.65:45147)
I20261119 09:04:00.438596 31910 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "bdebe591f2aa4d98b1a538f011b398f3" candidate_uuid: "a82d2f5d912a42a7bea8465837b33149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad7b4b42523b489f8ad7142d774cf5b7"
I20261119 09:04:00.438635 32041 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "bdebe591f2aa4d98b1a538f011b398f3" candidate_uuid: "a82d2f5d912a42a7bea8465837b33149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a6c5a6e8b664dfa846f479be75116e4"
I20261119 09:04:00.438758 31910 raft_consensus.cc:3096] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7 [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:00.438758 32041 raft_consensus.cc:3096] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:00.440099 32041 raft_consensus.cc:2490] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a82d2f5d912a42a7bea8465837b33149 in term 1.
I20261119 09:04:00.440099 31910 raft_consensus.cc:2490] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a82d2f5d912a42a7bea8465837b33149 in term 1.
I20261119 09:04:00.440462 32108 leader_election.cc:304] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a82d2f5d912a42a7bea8465837b33149, ad7b4b42523b489f8ad7142d774cf5b7; no voters:
I20261119 09:04:00.440909 32249 raft_consensus.cc:2826] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [term 1 FOLLOWER]: Leader election won for term 1
I20261119 09:04:00.441360 32249 raft_consensus.cc:705] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [term 1 LEADER]: Becoming Leader. State: Replica: a82d2f5d912a42a7bea8465837b33149, State: Running, Role: LEADER
I20261119 09:04:00.441781 32249 consensus_queue.cc:238] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } } peers { permanent_uuid: "ad7b4b42523b489f8ad7142d774cf5b7" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 45147 } }
I20261119 09:04:00.446544 31768 catalog_manager.cc:5856] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 reported cstate change: term changed from 0 to 1, leader changed from <none> to a82d2f5d912a42a7bea8465837b33149 (127.30.252.67). New cstate: current_term: 1 leader_uuid: "a82d2f5d912a42a7bea8465837b33149" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad7b4b42523b489f8ad7142d774cf5b7" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 45147 } health_report { overall_health: UNKNOWN } } }
I20261119 09:04:00.446837 32249 consensus_queue.cc:1049] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261119 09:04:00.447201 32254 consensus_queue.cc:1049] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad7b4b42523b489f8ad7142d774cf5b7" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20261119 09:04:00.650882 31950 generic_service.cc:139] {username='slave'} at 127.0.0.1:46570 forcing change of non-runtime-safe flag follower_reject_update_consensus_requests
I20261119 09:04:00.651021 31950 generic_service.cc:175] {username='slave'} at 127.0.0.1:46570 changed flags via RPC: follower_reject_update_consensus_requests from 'false' to 'false'
I20261119 09:04:00.652117 31950 generic_service.cc:175] {username='slave'} at 127.0.0.1:46570 changed flags via RPC: heartbeat_inject_latency_before_heartbeat_ms from '0' to '5000'
W20261119 09:04:00.653524 32081 generic_service.cc:139] {username='slave'} at 127.0.0.1:48938 forcing change of non-runtime-safe flag follower_reject_update_consensus_requests
I20261119 09:04:00.653824 32081 generic_service.cc:175] {username='slave'} at 127.0.0.1:48938 changed flags via RPC: follower_reject_update_consensus_requests from 'false' to 'false'
I20261119 09:04:00.655167 32081 generic_service.cc:175] {username='slave'} at 127.0.0.1:48938 changed flags via RPC: heartbeat_inject_latency_before_heartbeat_ms from '0' to '5000'
W20261119 09:04:00.656391 32212 generic_service.cc:139] {username='slave'} at 127.0.0.1:36606 forcing change of non-runtime-safe flag follower_reject_update_consensus_requests
I20261119 09:04:00.656517 32212 generic_service.cc:175] {username='slave'} at 127.0.0.1:36606 changed flags via RPC: follower_reject_update_consensus_requests from 'false' to 'false'
I20261119 09:04:00.657850 32212 generic_service.cc:175] {username='slave'} at 127.0.0.1:36606 changed flags via RPC: heartbeat_inject_latency_before_heartbeat_ms from '0' to '2000'
I20261119 09:04:00.660207 32172 consensus_queue.cc:238] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } }
I20261119 09:04:00.661522 32041 raft_consensus.cc:1285] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [term 1 FOLLOWER]: Refusing update from remote peer a82d2f5d912a42a7bea8465837b33149: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20261119 09:04:00.662168 32256 consensus_queue.cc:1049] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20261119 09:04:00.664776 32249 raft_consensus.cc:2983] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [term 1 LEADER]: Committing config change with OpId 1.24: config changed from index -1 to 24, VOTER ad7b4b42523b489f8ad7142d774cf5b7 (127.30.252.65) evicted. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } } }
I20261119 09:04:00.665294 32041 raft_consensus.cc:2983] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index -1 to 24, VOTER ad7b4b42523b489f8ad7142d774cf5b7 (127.30.252.65) evicted. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } } }
W20261119 09:04:00.667248 32212 generic_service.cc:139] {username='slave'} at 127.0.0.1:36606 forcing change of non-runtime-safe flag follower_reject_update_consensus_requests
I20261119 09:04:00.667312 32212 generic_service.cc:175] {username='slave'} at 127.0.0.1:36606 changed flags via RPC: follower_reject_update_consensus_requests from 'false' to 'true'
I20261119 09:04:00.668100 32041 tablet_service.cc:2028] Received Run Leader Election RPC: tablet_id: "bdebe591f2aa4d98b1a538f011b398f3"
dest_uuid: "5a6c5a6e8b664dfa846f479be75116e4"
from {username='slave'} at 127.0.0.1:48938
I20261119 09:04:00.668315 32041 raft_consensus.cc:499] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20261119 09:04:00.668396 32041 raft_consensus.cc:3096] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [term 1 FOLLOWER]: Advancing to term 2
I20261119 09:04:00.669206 32041 raft_consensus.cc:521] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } }
I20261119 09:04:00.670135 32041 leader_election.cc:290] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [CANDIDATE]: Term 2 election: Requested vote from peers a82d2f5d912a42a7bea8465837b33149 (127.30.252.67:44277)
I20261119 09:04:00.675326 32172 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "bdebe591f2aa4d98b1a538f011b398f3" candidate_uuid: "5a6c5a6e8b664dfa846f479be75116e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: true dest_uuid: "a82d2f5d912a42a7bea8465837b33149"
I20261119 09:04:00.675521 32172 raft_consensus.cc:3091] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [term 1 LEADER]: Stepping down as leader of term 1
I20261119 09:04:00.675609 32172 raft_consensus.cc:748] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a82d2f5d912a42a7bea8465837b33149, State: Running, Role: LEADER
I20261119 09:04:00.676067 32172 consensus_queue.cc:261] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } }
I20261119 09:04:00.676277 32172 raft_consensus.cc:3096] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [term 1 FOLLOWER]: Advancing to term 2
I20261119 09:04:00.677551 32172 raft_consensus.cc:2490] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a6c5a6e8b664dfa846f479be75116e4 in term 2.
I20261119 09:04:00.678727 31977 leader_election.cc:304] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5a6c5a6e8b664dfa846f479be75116e4, a82d2f5d912a42a7bea8465837b33149; no voters:
I20261119 09:04:00.679304 32258 raft_consensus.cc:2826] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [term 2 FOLLOWER]: Leader election won for term 2
I20261119 09:04:00.679555 32258 raft_consensus.cc:705] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [term 2 LEADER]: Becoming Leader. State: Replica: 5a6c5a6e8b664dfa846f479be75116e4, State: Running, Role: LEADER
I20261119 09:04:00.680186 32258 consensus_queue.cc:238] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } }
W20261119 09:04:00.685209 31977 consensus_peers.cc:597] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 -> Peer a82d2f5d912a42a7bea8465837b33149 (127.30.252.67:44277): Couldn't send request to peer a82d2f5d912a42a7bea8465837b33149. Error code: UNKNOWN_ERROR (1). Status: Illegal state: Rejected: --follower_reject_update_consensus_requests is set to true.. This is attempt 1: this message will repeat every 5th retry.
I20261119 09:04:02.667747 31768 catalog_manager.cc:5856] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 reported cstate change: config changed from index -1 to 24, term changed from 1 to 2, leader changed from a82d2f5d912a42a7bea8465837b33149 (127.30.252.67) to <none>, VOTER ad7b4b42523b489f8ad7142d774cf5b7 (127.30.252.65) evicted. New cstate: current_term: 2 committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } } }
I20261119 09:04:02.671722 31890 tablet_service.cc:1542] Processing DeleteTablet for tablet bdebe591f2aa4d98b1a538f011b398f3 with delete_type TABLET_DATA_TOMBSTONED (TS ad7b4b42523b489f8ad7142d774cf5b7 not found in new config with opid_index 24) from {username='slave'} at 127.0.0.1:46562
I20261119 09:04:02.672366 32293 tablet_replica.cc:333] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: stopping tablet replica
I20261119 09:04:02.672621 32293 raft_consensus.cc:2261] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261119 09:04:02.672856 32293 raft_consensus.cc:2290] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261119 09:04:02.673904 32293 ts_tablet_manager.cc:1962] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20261119 09:04:02.678063 32293 ts_tablet_manager.cc:1975] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.23
I20261119 09:04:02.678164 32293 log.cc:1199] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: Deleting WAL directory at /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795079038760673-31729-0/minicluster-data/ts-0/wal/wals/bdebe591f2aa4d98b1a538f011b398f3
I20261119 09:04:02.678941 31755 catalog_manager.cc:5081] TS ad7b4b42523b489f8ad7142d774cf5b7 (127.30.252.65:45147): tablet bdebe591f2aa4d98b1a538f011b398f3 (table test-workload [id=53f572791d434587895f156ca794a324]) successfully deleted
W20261119 09:04:02.720093 32212 generic_service.cc:139] {username='slave'} at 127.0.0.1:36606 forcing change of non-runtime-safe flag follower_reject_update_consensus_requests
I20261119 09:04:02.720192 32212 generic_service.cc:175] {username='slave'} at 127.0.0.1:36606 changed flags via RPC: follower_reject_update_consensus_requests from 'true' to 'false'
I20261119 09:04:02.725134 31729 cluster_itest_util.cc:258] Not converged past 1 yet: 2.25 1.24 <uninitialized op>
I20261119 09:04:02.828611 31729 cluster_itest_util.cc:258] Not converged past 1 yet: 2.25 1.24 <uninitialized op>
I20261119 09:04:03.032160 31729 cluster_itest_util.cc:258] Not converged past 1 yet: 2.25 1.24 <uninitialized op>
I20261119 09:04:03.336179 31729 cluster_itest_util.cc:258] Not converged past 1 yet: 2.25 1.24 <uninitialized op>
I20261119 09:04:03.739475 31729 cluster_itest_util.cc:258] Not converged past 1 yet: 2.25 1.24 <uninitialized op>
I20261119 09:04:04.244388 31729 cluster_itest_util.cc:258] Not converged past 1 yet: 2.25 1.24 <uninitialized op>
I20261119 09:04:04.673285 31768 catalog_manager.cc:5856] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 reported cstate change: leader changed from <none> to 5a6c5a6e8b664dfa846f479be75116e4 (127.30.252.66). New cstate: current_term: 2 leader_uuid: "5a6c5a6e8b664dfa846f479be75116e4" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } } }
I20261119 09:04:04.847659 31729 cluster_itest_util.cc:258] Not converged past 1 yet: 2.25 2.25 <uninitialized op>
I20261119 09:04:05.550655 31729 cluster_itest_util.cc:258] Not converged past 1 yet: 2.25 2.25 <uninitialized op>
I20261119 09:04:05.670867 32041 consensus_queue.cc:238] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 24, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } } peers { permanent_uuid: "ad7b4b42523b489f8ad7142d774cf5b7" member_type: NON_VOTER last_known_addr { host: "127.30.252.65" port: 45147 } attrs { promote: true } }
I20261119 09:04:05.672806 32172 raft_consensus.cc:1285] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [term 2 FOLLOWER]: Refusing update from remote peer 5a6c5a6e8b664dfa846f479be75116e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20261119 09:04:05.673301 32307 consensus_queue.cc:1049] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20261119 09:04:05.675855 32306 raft_consensus.cc:2983] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 [term 2 LEADER]: Committing config change with OpId 2.26: config changed from index 24 to 26, NON_VOTER ad7b4b42523b489f8ad7142d774cf5b7 (127.30.252.65) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } } peers { permanent_uuid: "ad7b4b42523b489f8ad7142d774cf5b7" member_type: NON_VOTER last_known_addr { host: "127.30.252.65" port: 45147 } attrs { promote: true } } }
I20261119 09:04:05.676252 32172 raft_consensus.cc:2983] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index 24 to 26, NON_VOTER ad7b4b42523b489f8ad7142d774cf5b7 (127.30.252.65) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } } peers { permanent_uuid: "ad7b4b42523b489f8ad7142d774cf5b7" member_type: NON_VOTER last_known_addr { host: "127.30.252.65" port: 45147 } attrs { promote: true } } }
W20261119 09:04:05.676491 31977 consensus_peers.cc:597] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4 -> Peer ad7b4b42523b489f8ad7142d774cf5b7 (127.30.252.65:45147): Couldn't send request to peer ad7b4b42523b489f8ad7142d774cf5b7. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20261119 09:04:05.677788 31753 catalog_manager.cc:5263] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bdebe591f2aa4d98b1a538f011b398f3 with cas_config_opid_index 24: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20261119 09:04:06.353827 31729 cluster_itest_util.cc:258] Not converged past 1 yet: 2.26 2.26 <uninitialized op>
I20261119 09:04:06.679183 31768 catalog_manager.cc:5856] T bdebe591f2aa4d98b1a538f011b398f3 P a82d2f5d912a42a7bea8465837b33149 reported cstate change: config changed from index 24 to 26, NON_VOTER ad7b4b42523b489f8ad7142d774cf5b7 (127.30.252.65) added. New cstate: current_term: 2 leader_uuid: "5a6c5a6e8b664dfa846f479be75116e4" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } } peers { permanent_uuid: "ad7b4b42523b489f8ad7142d774cf5b7" member_type: NON_VOTER last_known_addr { host: "127.30.252.65" port: 45147 } attrs { promote: true } } }
I20261119 09:04:07.257318 31729 cluster_itest_util.cc:258] Not converged past 1 yet: 2.26 2.26 <uninitialized op>
I20261119 09:04:08.260893 31729 cluster_itest_util.cc:258] Not converged past 1 yet: 2.26 2.26 <uninitialized op>
I20261119 09:04:09.077649 32319 ts_tablet_manager.cc:979] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: Initiating tablet copy from peer 5a6c5a6e8b664dfa846f479be75116e4 (127.30.252.66:46023)
I20261119 09:04:09.078315 32319 tablet_copy_client.cc:323] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: tablet copy: Beginning tablet copy session from remote peer at address 127.30.252.66:46023
I20261119 09:04:09.082150 32061 tablet_copy_service.cc:140] P 5a6c5a6e8b664dfa846f479be75116e4: Received BeginTabletCopySession request for tablet bdebe591f2aa4d98b1a538f011b398f3 from peer ad7b4b42523b489f8ad7142d774cf5b7 ({username='slave'} at 127.30.252.65:36757)
I20261119 09:04:09.082278 32061 tablet_copy_service.cc:161] P 5a6c5a6e8b664dfa846f479be75116e4: Beginning new tablet copy session on tablet bdebe591f2aa4d98b1a538f011b398f3 from peer ad7b4b42523b489f8ad7142d774cf5b7 at {username='slave'} at 127.30.252.65:36757: session id = ad7b4b42523b489f8ad7142d774cf5b7-bdebe591f2aa4d98b1a538f011b398f3
I20261119 09:04:09.083419 32061 tablet_copy_source_session.cc:215] T bdebe591f2aa4d98b1a538f011b398f3 P 5a6c5a6e8b664dfa846f479be75116e4: Tablet Copy: opened 0 blocks and 1 log segments
I20261119 09:04:09.084811 32319 ts_tablet_manager.cc:1962] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: Deleting tablet data with delete state TABLET_DATA_COPYING
I20261119 09:04:09.086988 32319 ts_tablet_manager.cc:1975] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.23
I20261119 09:04:09.087177 32319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdebe591f2aa4d98b1a538f011b398f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:09.088270 32319 tablet_copy_client.cc:806] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: tablet copy: Starting download of 0 data blocks...
I20261119 09:04:09.088409 32319 tablet_copy_client.cc:670] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: tablet copy: Starting download of 1 WAL segments...
I20261119 09:04:09.090437 32319 tablet_copy_client.cc:538] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261119 09:04:09.091864 32319 tablet_bootstrap.cc:492] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: Bootstrap starting.
I20261119 09:04:09.130170 32319 tablet_bootstrap.cc:492] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=1100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261119 09:04:09.130455 32319 tablet_bootstrap.cc:492] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: Bootstrap complete.
I20261119 09:04:09.130600 32319 ts_tablet_manager.cc:1449] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: Time spent bootstrapping tablet: real 0.039s user 0.036s sys 0.000s
I20261119 09:04:09.131042 32319 raft_consensus.cc:360] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } } peers { permanent_uuid: "ad7b4b42523b489f8ad7142d774cf5b7" member_type: NON_VOTER last_known_addr { host: "127.30.252.65" port: 45147 } attrs { promote: true } }
I20261119 09:04:09.131141 32319 raft_consensus.cc:748] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ad7b4b42523b489f8ad7142d774cf5b7, State: Initialized, Role: LEARNER
I20261119 09:04:09.131227 32319 consensus_queue.cc:261] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "5a6c5a6e8b664dfa846f479be75116e4" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 46023 } } peers { permanent_uuid: "a82d2f5d912a42a7bea8465837b33149" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44277 } } peers { permanent_uuid: "ad7b4b42523b489f8ad7142d774cf5b7" member_type: NON_VOTER last_known_addr { host: "127.30.252.65" port: 45147 } attrs { promote: true } }
I20261119 09:04:09.131740 32319 ts_tablet_manager.cc:1480] T bdebe591f2aa4d98b1a538f011b398f3 P ad7b4b42523b489f8ad7142d774cf5b7: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20261119 09:04:09.132429 32061 tablet_copy_service.cc:342] P 5a6c5a6e8b664dfa846f479be75116e4: Request end of tablet copy session ad7b4b42523b489f8ad7142d774cf5b7-bdebe591f2aa4d98b1a538f011b398f3 received from {username='slave'} at 127.30.252.65:36757
I20261119 09:04:09.132526 32061 tablet_copy_service.cc:434] P 5a6c5a6e8b664dfa846f479be75116e4: ending tablet copy session ad7b4b42523b489f8ad7142d774cf5b7-bdebe591f2aa4d98b1a538f011b398f3 on tablet bdebe591f2aa4d98b1a538f011b398f3 with peer ad7b4b42523b489f8ad7142d774cf5b7
I20261119 09:04:09.266465 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 31828
I20261119 09:04:09.272671 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 31959
I20261119 09:04:09.279012 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 32090
I20261119 09:04:09.285023 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 31737
2026-11-19T09:04:09Z chronyd exiting
[ OK ] RaftConfigChangeITest.TestKudu2147 (10536 ms)
[ RUN ] RaftConfigChangeITest.TestNonVoterPromotion
2026-11-19T09:04:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-11-19T09:04:09Z Disabled control of system clock
I20261119 09:04:09.314462 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.252.126:46059
--webserver_interface=127.30.252.126
--webserver_port=0
--builtin_ntp_servers=127.30.252.84:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.252.126:46059
--raft_prepare_replacement_before_eviction=true
--catalog_manager_evict_excess_replicas=false with env {}
W20261119 09:04:09.415952 32328 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:09.416131 32328 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:09.416172 32328 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:09.420186 32328 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261119 09:04:09.420256 32328 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:09.420284 32328 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261119 09:04:09.420305 32328 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261119 09:04:09.424597 32328 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/master-0/wal
--catalog_manager_evict_excess_replicas=false
--ipki_ca_key_size=768
--master_addresses=127.30.252.126:46059
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.252.126:46059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.252.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.32328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:09.425475 32328 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:09.426185 32328 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:09.431435 32334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:09.431434 32336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:09.431449 32333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:09.431686 32328 server_base.cc:1061] running on GCE node
I20261119 09:04:09.432094 32328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:09.432775 32328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:09.433933 32328 hybrid_clock.cc:648] HybridClock initialized: now 1795079049433910 us; error 34 us; skew 500 ppm
I20261119 09:04:09.435773 32328 webserver.cc:577] Webserver started at http://127.30.252.126:43941/ using document root <none> and password file <none>
I20261119 09:04:09.436342 32328 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:09.436393 32328 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:09.436689 32328 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:09.437996 32328 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/master-0/data/instance:
uuid: "0bb32f230a214ec38476f1033a9d2446"
format_stamp: "Formatted at 2026-11-19 09:04:09 on dist-test-slave-1fcg"
I20261119 09:04:09.438400 32328 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/master-0/wal/instance:
uuid: "0bb32f230a214ec38476f1033a9d2446"
format_stamp: "Formatted at 2026-11-19 09:04:09 on dist-test-slave-1fcg"
I20261119 09:04:09.441586 32328 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261119 09:04:09.443943 32342 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:09.445104 32328 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261119 09:04:09.445240 32328 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/master-0/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/master-0/wal
uuid: "0bb32f230a214ec38476f1033a9d2446"
format_stamp: "Formatted at 2026-11-19 09:04:09 on dist-test-slave-1fcg"
I20261119 09:04:09.445322 32328 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:09.455094 32328 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:09.455552 32328 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:09.455716 32328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:09.462352 32328 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.126:46059
I20261119 09:04:09.462363 32394 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.126:46059 every 8 connection(s)
I20261119 09:04:09.463719 32328 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/master-0/data/info.pb
I20261119 09:04:09.467051 32395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:09.469553 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 32328
I20261119 09:04:09.469646 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/master-0/wal/instance
I20261119 09:04:09.472671 32395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446: Bootstrap starting.
I20261119 09:04:09.475289 32395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:09.476279 32395 log.cc:826] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:09.477766 32395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446: No bootstrap required, opened a new log
I20261119 09:04:09.480373 32395 raft_consensus.cc:360] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb32f230a214ec38476f1033a9d2446" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 46059 } }
I20261119 09:04:09.480559 32395 raft_consensus.cc:386] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:04:09.480608 32395 raft_consensus.cc:748] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb32f230a214ec38476f1033a9d2446, State: Initialized, Role: FOLLOWER
I20261119 09:04:09.481016 32395 consensus_queue.cc:261] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb32f230a214ec38476f1033a9d2446" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 46059 } }
I20261119 09:04:09.481146 32395 raft_consensus.cc:400] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261119 09:04:09.481190 32395 raft_consensus.cc:499] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261119 09:04:09.481253 32395 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446 [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:09.481988 32395 raft_consensus.cc:521] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb32f230a214ec38476f1033a9d2446" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 46059 } }
I20261119 09:04:09.482313 32395 leader_election.cc:304] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bb32f230a214ec38476f1033a9d2446; no voters:
I20261119 09:04:09.482570 32395 leader_election.cc:290] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261119 09:04:09.482758 32400 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446 [term 1 FOLLOWER]: Leader election won for term 1
I20261119 09:04:09.483142 32400 raft_consensus.cc:705] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446 [term 1 LEADER]: Becoming Leader. State: Replica: 0bb32f230a214ec38476f1033a9d2446, State: Running, Role: LEADER
I20261119 09:04:09.483474 32400 consensus_queue.cc:238] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb32f230a214ec38476f1033a9d2446" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 46059 } }
I20261119 09:04:09.483557 32395 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446 [sys.catalog]: configured and running, proceeding with master startup.
I20261119 09:04:09.485435 32402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bb32f230a214ec38476f1033a9d2446. Latest consensus state: current_term: 1 leader_uuid: "0bb32f230a214ec38476f1033a9d2446" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb32f230a214ec38476f1033a9d2446" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 46059 } } }
I20261119 09:04:09.485569 32402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446 [sys.catalog]: This master's current role is: LEADER
I20261119 09:04:09.485891 32401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bb32f230a214ec38476f1033a9d2446" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb32f230a214ec38476f1033a9d2446" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 46059 } } }
I20261119 09:04:09.486016 32401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446 [sys.catalog]: This master's current role is: LEADER
I20261119 09:04:09.486124 32414 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261119 09:04:09.488862 32414 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261119 09:04:09.494060 32414 catalog_manager.cc:1398] Generated new cluster ID: 23aa611761a04afd91b79320bd6eab1a
I20261119 09:04:09.494131 32414 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261119 09:04:09.501968 32414 catalog_manager.cc:1421] Generated new certificate authority record
I20261119 09:04:09.503057 32414 catalog_manager.cc:1555] Loading token signing keys...
I20261119 09:04:09.511224 32414 catalog_manager.cc:6237] T 00000000000000000000000000000000 P 0bb32f230a214ec38476f1033a9d2446: Generated new TSK 0
I20261119 09:04:09.511945 32414 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261119 09:04:09.522090 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.65:0
--local_ip_for_outbound_sockets=127.30.252.65
--webserver_interface=127.30.252.65
--webserver_port=0
--tserver_master_addrs=127.30.252.126:46059
--builtin_ntp_servers=127.30.252.84:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20261119 09:04:09.631471 32419 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:09.631683 32419 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:09.631732 32419 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:09.635703 32419 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:09.635841 32419 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.65
I20261119 09:04:09.640422 32419 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.252.65
--webserver_port=0
--tserver_master_addrs=127.30.252.126:46059
--never_fsync=true
--heap_profile_path=/tmp/kudu.32419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:09.641275 32419 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:09.642058 32419 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:09.648393 32425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:09.648424 32424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:09.648411 32427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:09.648545 32419 server_base.cc:1061] running on GCE node
I20261119 09:04:09.649171 32419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:09.649860 32419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:09.651013 32419 hybrid_clock.cc:648] HybridClock initialized: now 1795079049650992 us; error 41 us; skew 500 ppm
I20261119 09:04:09.653172 32419 webserver.cc:577] Webserver started at http://127.30.252.65:42413/ using document root <none> and password file <none>
I20261119 09:04:09.653896 32419 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:09.653963 32419 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:09.654196 32419 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:09.656023 32419 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-0/data/instance:
uuid: "94729140864d471f9c0f33bda5960774"
format_stamp: "Formatted at 2026-11-19 09:04:09 on dist-test-slave-1fcg"
I20261119 09:04:09.656527 32419 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-0/wal/instance:
uuid: "94729140864d471f9c0f33bda5960774"
format_stamp: "Formatted at 2026-11-19 09:04:09 on dist-test-slave-1fcg"
I20261119 09:04:09.660408 32419 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20261119 09:04:09.663061 32433 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:09.664589 32419 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20261119 09:04:09.664702 32419 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-0/wal
uuid: "94729140864d471f9c0f33bda5960774"
format_stamp: "Formatted at 2026-11-19 09:04:09 on dist-test-slave-1fcg"
I20261119 09:04:09.664846 32419 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:09.676527 32419 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:09.676952 32419 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:09.677083 32419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:09.677546 32419 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:09.678802 32419 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:09.678856 32419 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:09.678915 32419 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:09.678952 32419 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:09.689289 32419 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.65:36465
I20261119 09:04:09.689321 32546 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.65:36465 every 8 connection(s)
I20261119 09:04:09.690564 32419 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-0/data/info.pb
I20261119 09:04:09.698066 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 32419
I20261119 09:04:09.698179 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-0/wal/instance
I20261119 09:04:09.699090 32547 heartbeater.cc:345] Connected to a master server at 127.30.252.126:46059
I20261119 09:04:09.699322 32547 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:09.699879 32547 heartbeater.cc:508] Master 127.30.252.126:46059 requested a full tablet report, sending...
I20261119 09:04:09.700197 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.66:0
--local_ip_for_outbound_sockets=127.30.252.66
--webserver_interface=127.30.252.66
--webserver_port=0
--tserver_master_addrs=127.30.252.126:46059
--builtin_ntp_servers=127.30.252.84:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20261119 09:04:09.701490 32355 ts_manager.cc:210] Registered new tserver with Master: 94729140864d471f9c0f33bda5960774 (127.30.252.65:36465)
I20261119 09:04:09.702761 32355 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.65:59643
W20261119 09:04:09.808022 32550 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:09.808195 32550 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:09.808243 32550 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:09.811975 32550 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:09.812106 32550 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.66
I20261119 09:04:09.816618 32550 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.252.66
--webserver_port=0
--tserver_master_addrs=127.30.252.126:46059
--never_fsync=true
--heap_profile_path=/tmp/kudu.32550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:09.817742 32550 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:09.818526 32550 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:09.827458 32556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:09.827628 32550 server_base.cc:1061] running on GCE node
W20261119 09:04:09.827419 32555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:09.827390 32558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:09.828256 32550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:09.828892 32550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:09.830097 32550 hybrid_clock.cc:648] HybridClock initialized: now 1795079049830079 us; error 32 us; skew 500 ppm
I20261119 09:04:09.832829 32550 webserver.cc:577] Webserver started at http://127.30.252.66:40159/ using document root <none> and password file <none>
I20261119 09:04:09.833635 32550 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:09.833698 32550 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:09.833993 32550 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:09.835839 32550 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-1/data/instance:
uuid: "71f1d121e6c747b3b9742676d4660781"
format_stamp: "Formatted at 2026-11-19 09:04:09 on dist-test-slave-1fcg"
I20261119 09:04:09.836396 32550 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-1/wal/instance:
uuid: "71f1d121e6c747b3b9742676d4660781"
format_stamp: "Formatted at 2026-11-19 09:04:09 on dist-test-slave-1fcg"
I20261119 09:04:09.840866 32550 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20261119 09:04:09.844257 32564 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:09.845498 32550 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20261119 09:04:09.845638 32550 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-1/wal
uuid: "71f1d121e6c747b3b9742676d4660781"
format_stamp: "Formatted at 2026-11-19 09:04:09 on dist-test-slave-1fcg"
I20261119 09:04:09.845747 32550 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:09.869742 32550 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:09.870235 32550 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:09.870374 32550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:09.871026 32550 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:09.872116 32550 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:09.872166 32550 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:09.872224 32550 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:09.872254 32550 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:09.883490 32550 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.66:42007
I20261119 09:04:09.883502 32677 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.66:42007 every 8 connection(s)
I20261119 09:04:09.884618 32550 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-1/data/info.pb
I20261119 09:04:09.887246 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 32550
I20261119 09:04:09.887356 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-1/wal/instance
I20261119 09:04:09.890069 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.67:0
--local_ip_for_outbound_sockets=127.30.252.67
--webserver_interface=127.30.252.67
--webserver_port=0
--tserver_master_addrs=127.30.252.126:46059
--builtin_ntp_servers=127.30.252.84:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20261119 09:04:09.893666 32678 heartbeater.cc:345] Connected to a master server at 127.30.252.126:46059
I20261119 09:04:09.893904 32678 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:09.894397 32678 heartbeater.cc:508] Master 127.30.252.126:46059 requested a full tablet report, sending...
I20261119 09:04:09.896070 32355 ts_manager.cc:210] Registered new tserver with Master: 71f1d121e6c747b3b9742676d4660781 (127.30.252.66:42007)
I20261119 09:04:09.896730 32355 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.66:51921
W20261119 09:04:09.998607 32681 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:09.998847 32681 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:09.998914 32681 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:10.002553 32681 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:10.002688 32681 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.67
I20261119 09:04:10.006915 32681 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.252.67
--webserver_port=0
--tserver_master_addrs=127.30.252.126:46059
--never_fsync=true
--heap_profile_path=/tmp/kudu.32681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:10.007778 32681 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:10.008481 32681 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:10.014757 32686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:10.014762 32687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:10.014813 32689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:10.016135 32681 server_base.cc:1061] running on GCE node
I20261119 09:04:10.016572 32681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:10.017201 32681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:10.018375 32681 hybrid_clock.cc:648] HybridClock initialized: now 1795079050018324 us; error 70 us; skew 500 ppm
I20261119 09:04:10.021519 32681 webserver.cc:577] Webserver started at http://127.30.252.67:41807/ using document root <none> and password file <none>
I20261119 09:04:10.022390 32681 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:10.022473 32681 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:10.022722 32681 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:10.024232 32681 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-2/data/instance:
uuid: "336997efdf5446f5b29eadd804becaa9"
format_stamp: "Formatted at 2026-11-19 09:04:10 on dist-test-slave-1fcg"
I20261119 09:04:10.024827 32681 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-2/wal/instance:
uuid: "336997efdf5446f5b29eadd804becaa9"
format_stamp: "Formatted at 2026-11-19 09:04:10 on dist-test-slave-1fcg"
I20261119 09:04:10.028820 32681 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20261119 09:04:10.031396 32695 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:10.032507 32681 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261119 09:04:10.032614 32681 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-2/wal
uuid: "336997efdf5446f5b29eadd804becaa9"
format_stamp: "Formatted at 2026-11-19 09:04:10 on dist-test-slave-1fcg"
I20261119 09:04:10.032702 32681 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:10.054291 32681 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:10.054754 32681 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:10.054886 32681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:10.055365 32681 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:10.057126 32681 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:10.057379 32681 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:10.057494 32681 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:10.057528 32681 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:10.067504 32681 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.67:42461
I20261119 09:04:10.067517 340 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.67:42461 every 8 connection(s)
I20261119 09:04:10.068655 32681 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-2/data/info.pb
I20261119 09:04:10.076399 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 32681
I20261119 09:04:10.076531 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-2/wal/instance
I20261119 09:04:10.077176 341 heartbeater.cc:345] Connected to a master server at 127.30.252.126:46059
I20261119 09:04:10.077431 341 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:10.078053 341 heartbeater.cc:508] Master 127.30.252.126:46059 requested a full tablet report, sending...
I20261119 09:04:10.078969 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.68:0
--local_ip_for_outbound_sockets=127.30.252.68
--webserver_interface=127.30.252.68
--webserver_port=0
--tserver_master_addrs=127.30.252.126:46059
--builtin_ntp_servers=127.30.252.84:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20261119 09:04:10.079310 32355 ts_manager.cc:210] Registered new tserver with Master: 336997efdf5446f5b29eadd804becaa9 (127.30.252.67:42461)
I20261119 09:04:10.079950 32355 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.67:37717
W20261119 09:04:10.183512 344 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:10.183703 344 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:10.183760 344 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:10.187348 344 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:10.187490 344 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.68
I20261119 09:04:10.191417 344 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.30.252.68
--webserver_port=0
--tserver_master_addrs=127.30.252.126:46059
--never_fsync=true
--heap_profile_path=/tmp/kudu.344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.68
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:10.192222 344 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:10.192898 344 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:10.199355 352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:10.199349 349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:10.199357 350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:10.199837 344 server_base.cc:1061] running on GCE node
I20261119 09:04:10.200181 344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:10.200729 344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:10.201915 344 hybrid_clock.cc:648] HybridClock initialized: now 1795079050201895 us; error 33 us; skew 500 ppm
I20261119 09:04:10.204284 344 webserver.cc:577] Webserver started at http://127.30.252.68:44261/ using document root <none> and password file <none>
I20261119 09:04:10.204887 344 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:10.204938 344 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:10.205164 344 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:10.206470 344 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-3/data/instance:
uuid: "138c2912f98648ff9da32e725733bec7"
format_stamp: "Formatted at 2026-11-19 09:04:10 on dist-test-slave-1fcg"
I20261119 09:04:10.206897 344 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-3/wal/instance:
uuid: "138c2912f98648ff9da32e725733bec7"
format_stamp: "Formatted at 2026-11-19 09:04:10 on dist-test-slave-1fcg"
I20261119 09:04:10.210274 344 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20261119 09:04:10.212862 358 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:10.213904 344 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261119 09:04:10.214002 344 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-3/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-3/wal
uuid: "138c2912f98648ff9da32e725733bec7"
format_stamp: "Formatted at 2026-11-19 09:04:10 on dist-test-slave-1fcg"
I20261119 09:04:10.214077 344 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:10.236918 344 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:10.237365 344 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:10.237494 344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:10.237938 344 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:10.238888 344 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:10.238938 344 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:10.238983 344 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:10.238997 344 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:10.247797 344 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.68:42753
I20261119 09:04:10.247841 471 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.68:42753 every 8 connection(s)
I20261119 09:04:10.248838 344 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-3/data/info.pb
I20261119 09:04:10.256245 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 344
I20261119 09:04:10.256408 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795079038760673-31729-0/minicluster-data/ts-3/wal/instance
I20261119 09:04:10.256989 472 heartbeater.cc:345] Connected to a master server at 127.30.252.126:46059
I20261119 09:04:10.257241 472 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:10.257833 472 heartbeater.cc:508] Master 127.30.252.126:46059 requested a full tablet report, sending...
I20261119 09:04:10.258883 32359 ts_manager.cc:210] Registered new tserver with Master: 138c2912f98648ff9da32e725733bec7 (127.30.252.68:42753)
I20261119 09:04:10.259471 32359 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.68:57499
I20261119 09:04:10.271272 31729 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261119 09:04:10.292577 32359 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56706:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261119 09:04:10.312285 32481 tablet_service.cc:1495] Processing CreateTablet for tablet 214033a1727e4c7fb2602f438965766c (DEFAULT_TABLE table=test-workload [id=c5020ca1dab541c0b346e3ad8928ea43]), partition=RANGE (key) PARTITION UNBOUNDED
I20261119 09:04:10.312654 406 tablet_service.cc:1495] Processing CreateTablet for tablet 214033a1727e4c7fb2602f438965766c (DEFAULT_TABLE table=test-workload [id=c5020ca1dab541c0b346e3ad8928ea43]), partition=RANGE (key) PARTITION UNBOUNDED
I20261119 09:04:10.312654 32743 tablet_service.cc:1495] Processing CreateTablet for tablet 214033a1727e4c7fb2602f438965766c (DEFAULT_TABLE table=test-workload [id=c5020ca1dab541c0b346e3ad8928ea43]), partition=RANGE (key) PARTITION UNBOUNDED
I20261119 09:04:10.313285 32481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 214033a1727e4c7fb2602f438965766c. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:10.313573 32743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 214033a1727e4c7fb2602f438965766c. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:10.313573 406 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 214033a1727e4c7fb2602f438965766c. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:10.318593 494 tablet_bootstrap.cc:492] T 214033a1727e4c7fb2602f438965766c P 94729140864d471f9c0f33bda5960774: Bootstrap starting.
I20261119 09:04:10.320788 496 tablet_bootstrap.cc:492] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7: Bootstrap starting.
I20261119 09:04:10.321663 494 tablet_bootstrap.cc:654] T 214033a1727e4c7fb2602f438965766c P 94729140864d471f9c0f33bda5960774: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:10.322675 494 log.cc:826] T 214033a1727e4c7fb2602f438965766c P 94729140864d471f9c0f33bda5960774: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:10.323189 496 tablet_bootstrap.cc:654] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:10.323225 495 tablet_bootstrap.cc:492] T 214033a1727e4c7fb2602f438965766c P 336997efdf5446f5b29eadd804becaa9: Bootstrap starting.
I20261119 09:04:10.324081 496 log.cc:826] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:10.324415 494 tablet_bootstrap.cc:492] T 214033a1727e4c7fb2602f438965766c P 94729140864d471f9c0f33bda5960774: No bootstrap required, opened a new log
I20261119 09:04:10.324623 494 ts_tablet_manager.cc:1449] T 214033a1727e4c7fb2602f438965766c P 94729140864d471f9c0f33bda5960774: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20261119 09:04:10.325379 495 tablet_bootstrap.cc:654] T 214033a1727e4c7fb2602f438965766c P 336997efdf5446f5b29eadd804becaa9: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:10.325526 496 tablet_bootstrap.cc:492] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7: No bootstrap required, opened a new log
I20261119 09:04:10.325754 496 ts_tablet_manager.cc:1449] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20261119 09:04:10.326359 495 log.cc:826] T 214033a1727e4c7fb2602f438965766c P 336997efdf5446f5b29eadd804becaa9: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:10.327710 494 raft_consensus.cc:360] T 214033a1727e4c7fb2602f438965766c P 94729140864d471f9c0f33bda5960774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } }
I20261119 09:04:10.327934 495 tablet_bootstrap.cc:492] T 214033a1727e4c7fb2602f438965766c P 336997efdf5446f5b29eadd804becaa9: No bootstrap required, opened a new log
I20261119 09:04:10.328011 494 raft_consensus.cc:386] T 214033a1727e4c7fb2602f438965766c P 94729140864d471f9c0f33bda5960774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:04:10.328109 494 raft_consensus.cc:748] T 214033a1727e4c7fb2602f438965766c P 94729140864d471f9c0f33bda5960774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94729140864d471f9c0f33bda5960774, State: Initialized, Role: FOLLOWER
I20261119 09:04:10.328118 495 ts_tablet_manager.cc:1449] T 214033a1727e4c7fb2602f438965766c P 336997efdf5446f5b29eadd804becaa9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20261119 09:04:10.328557 494 consensus_queue.cc:261] T 214033a1727e4c7fb2602f438965766c P 94729140864d471f9c0f33bda5960774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } }
I20261119 09:04:10.328706 496 raft_consensus.cc:360] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } }
I20261119 09:04:10.328938 496 raft_consensus.cc:386] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:04:10.329043 496 raft_consensus.cc:748] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138c2912f98648ff9da32e725733bec7, State: Initialized, Role: FOLLOWER
I20261119 09:04:10.329311 32547 heartbeater.cc:500] Master 127.30.252.126:46059 was elected leader, sending a full tablet report...
I20261119 09:04:10.329375 494 ts_tablet_manager.cc:1480] T 214033a1727e4c7fb2602f438965766c P 94729140864d471f9c0f33bda5960774: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20261119 09:04:10.329480 496 consensus_queue.cc:261] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } }
I20261119 09:04:10.330098 472 heartbeater.cc:500] Master 127.30.252.126:46059 was elected leader, sending a full tablet report...
I20261119 09:04:10.330245 496 ts_tablet_manager.cc:1480] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20261119 09:04:10.331068 495 raft_consensus.cc:360] T 214033a1727e4c7fb2602f438965766c P 336997efdf5446f5b29eadd804becaa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } }
I20261119 09:04:10.331260 495 raft_consensus.cc:386] T 214033a1727e4c7fb2602f438965766c P 336997efdf5446f5b29eadd804becaa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:04:10.331346 495 raft_consensus.cc:748] T 214033a1727e4c7fb2602f438965766c P 336997efdf5446f5b29eadd804becaa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 336997efdf5446f5b29eadd804becaa9, State: Initialized, Role: FOLLOWER
I20261119 09:04:10.331748 495 consensus_queue.cc:261] T 214033a1727e4c7fb2602f438965766c P 336997efdf5446f5b29eadd804becaa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } }
I20261119 09:04:10.332397 341 heartbeater.cc:500] Master 127.30.252.126:46059 was elected leader, sending a full tablet report...
I20261119 09:04:10.332412 495 ts_tablet_manager.cc:1480] T 214033a1727e4c7fb2602f438965766c P 336997efdf5446f5b29eadd804becaa9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20261119 09:04:10.437005 501 raft_consensus.cc:499] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261119 09:04:10.437304 501 raft_consensus.cc:521] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } }
I20261119 09:04:10.438669 501 leader_election.cc:290] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94729140864d471f9c0f33bda5960774 (127.30.252.65:36465), 336997efdf5446f5b29eadd804becaa9 (127.30.252.67:42461)
W20261119 09:04:10.441865 32548 tablet.cc:2554] T 214033a1727e4c7fb2602f438965766c P 94729140864d471f9c0f33bda5960774: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261119 09:04:10.442304 32501 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "214033a1727e4c7fb2602f438965766c" candidate_uuid: "138c2912f98648ff9da32e725733bec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94729140864d471f9c0f33bda5960774" is_pre_election: true
I20261119 09:04:10.442570 32501 raft_consensus.cc:2490] T 214033a1727e4c7fb2602f438965766c P 94729140864d471f9c0f33bda5960774 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 138c2912f98648ff9da32e725733bec7 in term 0.
I20261119 09:04:10.442634 32763 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "214033a1727e4c7fb2602f438965766c" candidate_uuid: "138c2912f98648ff9da32e725733bec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "336997efdf5446f5b29eadd804becaa9" is_pre_election: true
I20261119 09:04:10.442880 32763 raft_consensus.cc:2490] T 214033a1727e4c7fb2602f438965766c P 336997efdf5446f5b29eadd804becaa9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 138c2912f98648ff9da32e725733bec7 in term 0.
I20261119 09:04:10.443024 359 leader_election.cc:304] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138c2912f98648ff9da32e725733bec7, 94729140864d471f9c0f33bda5960774; no voters:
I20261119 09:04:10.443272 501 raft_consensus.cc:2826] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261119 09:04:10.443363 501 raft_consensus.cc:499] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261119 09:04:10.443414 501 raft_consensus.cc:3096] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:10.444422 501 raft_consensus.cc:521] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } }
I20261119 09:04:10.444741 501 leader_election.cc:290] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [CANDIDATE]: Term 1 election: Requested vote from peers 94729140864d471f9c0f33bda5960774 (127.30.252.65:36465), 336997efdf5446f5b29eadd804becaa9 (127.30.252.67:42461)
I20261119 09:04:10.445083 32763 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "214033a1727e4c7fb2602f438965766c" candidate_uuid: "138c2912f98648ff9da32e725733bec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "336997efdf5446f5b29eadd804becaa9"
I20261119 09:04:10.445138 32501 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "214033a1727e4c7fb2602f438965766c" candidate_uuid: "138c2912f98648ff9da32e725733bec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94729140864d471f9c0f33bda5960774"
I20261119 09:04:10.445192 32763 raft_consensus.cc:3096] T 214033a1727e4c7fb2602f438965766c P 336997efdf5446f5b29eadd804becaa9 [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:10.445263 32501 raft_consensus.cc:3096] T 214033a1727e4c7fb2602f438965766c P 94729140864d471f9c0f33bda5960774 [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:10.446060 32763 raft_consensus.cc:2490] T 214033a1727e4c7fb2602f438965766c P 336997efdf5446f5b29eadd804becaa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 138c2912f98648ff9da32e725733bec7 in term 1.
I20261119 09:04:10.446341 361 leader_election.cc:304] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138c2912f98648ff9da32e725733bec7, 336997efdf5446f5b29eadd804becaa9; no voters:
I20261119 09:04:10.446457 32501 raft_consensus.cc:2490] T 214033a1727e4c7fb2602f438965766c P 94729140864d471f9c0f33bda5960774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 138c2912f98648ff9da32e725733bec7 in term 1.
I20261119 09:04:10.446563 501 raft_consensus.cc:2826] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [term 1 FOLLOWER]: Leader election won for term 1
I20261119 09:04:10.446786 501 raft_consensus.cc:705] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [term 1 LEADER]: Becoming Leader. State: Replica: 138c2912f98648ff9da32e725733bec7, State: Running, Role: LEADER
I20261119 09:04:10.447139 501 consensus_queue.cc:238] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } }
I20261119 09:04:10.450443 504 consensus_queue.cc:1049] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261119 09:04:10.450337 32355 catalog_manager.cc:5856] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 138c2912f98648ff9da32e725733bec7 (127.30.252.68). New cstate: current_term: 1 leader_uuid: "138c2912f98648ff9da32e725733bec7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } health_report { overall_health: UNKNOWN } } }
I20261119 09:04:10.451131 504 consensus_queue.cc:1049] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261119 09:04:10.475883 426 consensus_queue.cc:238] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } } peers { permanent_uuid: "71f1d121e6c747b3b9742676d4660781" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 42007 } attrs { promote: true } }
I20261119 09:04:10.477422 32763 raft_consensus.cc:1285] T 214033a1727e4c7fb2602f438965766c P 336997efdf5446f5b29eadd804becaa9 [term 1 FOLLOWER]: Refusing update from remote peer 138c2912f98648ff9da32e725733bec7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20261119 09:04:10.477509 32501 raft_consensus.cc:1285] T 214033a1727e4c7fb2602f438965766c P 94729140864d471f9c0f33bda5960774 [term 1 FOLLOWER]: Refusing update from remote peer 138c2912f98648ff9da32e725733bec7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20261119 09:04:10.477905 504 consensus_queue.cc:1049] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261119 09:04:10.478107 506 consensus_queue.cc:1049] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261119 09:04:10.479853 501 raft_consensus.cc:2983] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 71f1d121e6c747b3b9742676d4660781 (127.30.252.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } } peers { permanent_uuid: "71f1d121e6c747b3b9742676d4660781" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 42007 } attrs { promote: true } } }
I20261119 09:04:10.480247 32763 raft_consensus.cc:2983] T 214033a1727e4c7fb2602f438965766c P 336997efdf5446f5b29eadd804becaa9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 71f1d121e6c747b3b9742676d4660781 (127.30.252.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } } peers { permanent_uuid: "71f1d121e6c747b3b9742676d4660781" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 42007 } attrs { promote: true } } }
I20261119 09:04:10.482172 32501 raft_consensus.cc:2983] T 214033a1727e4c7fb2602f438965766c P 94729140864d471f9c0f33bda5960774 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 71f1d121e6c747b3b9742676d4660781 (127.30.252.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } } peers { permanent_uuid: "71f1d121e6c747b3b9742676d4660781" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 42007 } attrs { promote: true } } }
W20261119 09:04:10.482764 362 consensus_peers.cc:597] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 -> Peer 71f1d121e6c747b3b9742676d4660781 (127.30.252.66:42007): Couldn't send request to peer 71f1d121e6c747b3b9742676d4660781. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 214033a1727e4c7fb2602f438965766c. This is attempt 1: this message will repeat every 5th retry.
I20261119 09:04:10.482820 32359 catalog_manager.cc:5856] T 214033a1727e4c7fb2602f438965766c P 336997efdf5446f5b29eadd804becaa9 reported cstate change: config changed from index -1 to 2, NON_VOTER 71f1d121e6c747b3b9742676d4660781 (127.30.252.66) added. New cstate: current_term: 1 leader_uuid: "138c2912f98648ff9da32e725733bec7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } } peers { permanent_uuid: "71f1d121e6c747b3b9742676d4660781" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 42007 } attrs { promote: true } } }
I20261119 09:04:10.899164 32678 heartbeater.cc:500] Master 127.30.252.126:46059 was elected leader, sending a full tablet report...
I20261119 09:04:11.038663 514 ts_tablet_manager.cc:979] T 214033a1727e4c7fb2602f438965766c P 71f1d121e6c747b3b9742676d4660781: Initiating tablet copy from peer 138c2912f98648ff9da32e725733bec7 (127.30.252.68:42753)
I20261119 09:04:11.039322 514 tablet_copy_client.cc:323] T 214033a1727e4c7fb2602f438965766c P 71f1d121e6c747b3b9742676d4660781: tablet copy: Beginning tablet copy session from remote peer at address 127.30.252.68:42753
I20261119 09:04:11.043339 446 tablet_copy_service.cc:140] P 138c2912f98648ff9da32e725733bec7: Received BeginTabletCopySession request for tablet 214033a1727e4c7fb2602f438965766c from peer 71f1d121e6c747b3b9742676d4660781 ({username='slave'} at 127.30.252.66:46283)
I20261119 09:04:11.043473 446 tablet_copy_service.cc:161] P 138c2912f98648ff9da32e725733bec7: Beginning new tablet copy session on tablet 214033a1727e4c7fb2602f438965766c from peer 71f1d121e6c747b3b9742676d4660781 at {username='slave'} at 127.30.252.66:46283: session id = 71f1d121e6c747b3b9742676d4660781-214033a1727e4c7fb2602f438965766c
I20261119 09:04:11.044730 446 tablet_copy_source_session.cc:215] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7: Tablet Copy: opened 0 blocks and 1 log segments
I20261119 09:04:11.048265 514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 214033a1727e4c7fb2602f438965766c. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:11.053207 514 tablet_copy_client.cc:806] T 214033a1727e4c7fb2602f438965766c P 71f1d121e6c747b3b9742676d4660781: tablet copy: Starting download of 0 data blocks...
I20261119 09:04:11.053399 514 tablet_copy_client.cc:670] T 214033a1727e4c7fb2602f438965766c P 71f1d121e6c747b3b9742676d4660781: tablet copy: Starting download of 1 WAL segments...
I20261119 09:04:11.055266 514 tablet_copy_client.cc:538] T 214033a1727e4c7fb2602f438965766c P 71f1d121e6c747b3b9742676d4660781: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261119 09:04:11.056953 514 tablet_bootstrap.cc:492] T 214033a1727e4c7fb2602f438965766c P 71f1d121e6c747b3b9742676d4660781: Bootstrap starting.
I20261119 09:04:11.061865 514 log.cc:826] T 214033a1727e4c7fb2602f438965766c P 71f1d121e6c747b3b9742676d4660781: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:11.066016 514 tablet_bootstrap.cc:492] T 214033a1727e4c7fb2602f438965766c P 71f1d121e6c747b3b9742676d4660781: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261119 09:04:11.066308 514 tablet_bootstrap.cc:492] T 214033a1727e4c7fb2602f438965766c P 71f1d121e6c747b3b9742676d4660781: Bootstrap complete.
I20261119 09:04:11.066607 514 ts_tablet_manager.cc:1449] T 214033a1727e4c7fb2602f438965766c P 71f1d121e6c747b3b9742676d4660781: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20261119 09:04:11.069105 514 raft_consensus.cc:360] T 214033a1727e4c7fb2602f438965766c P 71f1d121e6c747b3b9742676d4660781 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } } peers { permanent_uuid: "71f1d121e6c747b3b9742676d4660781" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 42007 } attrs { promote: true } }
I20261119 09:04:11.069316 514 raft_consensus.cc:748] T 214033a1727e4c7fb2602f438965766c P 71f1d121e6c747b3b9742676d4660781 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 71f1d121e6c747b3b9742676d4660781, State: Initialized, Role: LEARNER
I20261119 09:04:11.069638 514 consensus_queue.cc:261] T 214033a1727e4c7fb2602f438965766c P 71f1d121e6c747b3b9742676d4660781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } } peers { permanent_uuid: "71f1d121e6c747b3b9742676d4660781" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 42007 } attrs { promote: true } }
I20261119 09:04:11.070406 514 ts_tablet_manager.cc:1480] T 214033a1727e4c7fb2602f438965766c P 71f1d121e6c747b3b9742676d4660781: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20261119 09:04:11.071143 446 tablet_copy_service.cc:342] P 138c2912f98648ff9da32e725733bec7: Request end of tablet copy session 71f1d121e6c747b3b9742676d4660781-214033a1727e4c7fb2602f438965766c received from {username='slave'} at 127.30.252.66:46283
I20261119 09:04:11.071238 446 tablet_copy_service.cc:434] P 138c2912f98648ff9da32e725733bec7: ending tablet copy session 71f1d121e6c747b3b9742676d4660781-214033a1727e4c7fb2602f438965766c on tablet 214033a1727e4c7fb2602f438965766c with peer 71f1d121e6c747b3b9742676d4660781
I20261119 09:04:11.469678 32632 raft_consensus.cc:1227] T 214033a1727e4c7fb2602f438965766c P 71f1d121e6c747b3b9742676d4660781 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20261119 09:04:12.083595 521 raft_consensus.cc:1074] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7: attempting to promote NON_VOTER 71f1d121e6c747b3b9742676d4660781 to VOTER
I20261119 09:04:12.084070 521 consensus_queue.cc:238] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } } peers { permanent_uuid: "71f1d121e6c747b3b9742676d4660781" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 42007 } attrs { promote: false } }
I20261119 09:04:12.085392 32763 raft_consensus.cc:1285] T 214033a1727e4c7fb2602f438965766c P 336997efdf5446f5b29eadd804becaa9 [term 1 FOLLOWER]: Refusing update from remote peer 138c2912f98648ff9da32e725733bec7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20261119 09:04:12.085458 32632 raft_consensus.cc:1285] T 214033a1727e4c7fb2602f438965766c P 71f1d121e6c747b3b9742676d4660781 [term 1 LEARNER]: Refusing update from remote peer 138c2912f98648ff9da32e725733bec7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20261119 09:04:12.085559 32501 raft_consensus.cc:1285] T 214033a1727e4c7fb2602f438965766c P 94729140864d471f9c0f33bda5960774 [term 1 FOLLOWER]: Refusing update from remote peer 138c2912f98648ff9da32e725733bec7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20261119 09:04:12.085752 520 consensus_queue.cc:1049] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261119 09:04:12.085983 523 consensus_queue.cc:1049] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261119 09:04:12.086190 524 consensus_queue.cc:1049] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71f1d121e6c747b3b9742676d4660781" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 42007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261119 09:04:12.087770 524 raft_consensus.cc:2983] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 71f1d121e6c747b3b9742676d4660781 (127.30.252.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } } peers { permanent_uuid: "71f1d121e6c747b3b9742676d4660781" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 42007 } attrs { promote: false } } }
I20261119 09:04:12.088232 32501 raft_consensus.cc:2983] T 214033a1727e4c7fb2602f438965766c P 94729140864d471f9c0f33bda5960774 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 71f1d121e6c747b3b9742676d4660781 (127.30.252.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } } peers { permanent_uuid: "71f1d121e6c747b3b9742676d4660781" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 42007 } attrs { promote: false } } }
I20261119 09:04:12.089324 32632 raft_consensus.cc:2983] T 214033a1727e4c7fb2602f438965766c P 71f1d121e6c747b3b9742676d4660781 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 71f1d121e6c747b3b9742676d4660781 (127.30.252.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } } peers { permanent_uuid: "71f1d121e6c747b3b9742676d4660781" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 42007 } attrs { promote: false } } }
I20261119 09:04:12.090283 32355 catalog_manager.cc:5856] T 214033a1727e4c7fb2602f438965766c P 138c2912f98648ff9da32e725733bec7 reported cstate change: config changed from index 2 to 3, 71f1d121e6c747b3b9742676d4660781 (127.30.252.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "138c2912f98648ff9da32e725733bec7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71f1d121e6c747b3b9742676d4660781" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 42007 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20261119 09:04:12.091244 32763 raft_consensus.cc:2983] T 214033a1727e4c7fb2602f438965766c P 336997efdf5446f5b29eadd804becaa9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 71f1d121e6c747b3b9742676d4660781 (127.30.252.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "94729140864d471f9c0f33bda5960774" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 36465 } } peers { permanent_uuid: "138c2912f98648ff9da32e725733bec7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42753 } } peers { permanent_uuid: "336997efdf5446f5b29eadd804becaa9" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42461 } } peers { permanent_uuid: "71f1d121e6c747b3b9742676d4660781" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 42007 } attrs { promote: false } } }
I20261119 09:04:12.173256 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 32419
I20261119 09:04:12.179334 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 32550
I20261119 09:04:12.184710 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 32681
I20261119 09:04:12.189937 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 344
I20261119 09:04:12.194906 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 32328
2026-11-19T09:04:12Z chronyd exiting
[ OK ] RaftConfigChangeITest.TestNonVoterPromotion (2911 ms)
[ RUN ] RaftConfigChangeITest.TestBulkChangeConfig
2026-11-19T09:04:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-11-19T09:04:12Z Disabled control of system clock
I20261119 09:04:12.226663 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.252.126:39461
--webserver_interface=127.30.252.126
--webserver_port=0
--builtin_ntp_servers=127.30.252.84:44289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.252.126:39461
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--raft_prepare_replacement_before_eviction=true with env {}
W20261119 09:04:12.336480 538 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:12.336683 538 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:12.336724 538 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:12.340299 538 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261119 09:04:12.340359 538 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:12.340379 538 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261119 09:04:12.340396 538 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261119 09:04:12.344784 538 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:44289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.30.252.126:39461
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.252.126:39461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.252.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:12.345628 538 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:12.346359 538 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:12.351890 546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:12.351922 543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:12.352108 538 server_base.cc:1061] running on GCE node
W20261119 09:04:12.351899 544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:12.352690 538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:12.353509 538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:12.354691 538 hybrid_clock.cc:648] HybridClock initialized: now 1795079052354661 us; error 48 us; skew 500 ppm
I20261119 09:04:12.356596 538 webserver.cc:577] Webserver started at http://127.30.252.126:34789/ using document root <none> and password file <none>
I20261119 09:04:12.357189 538 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:12.357239 538 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:12.357465 538 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:12.358811 538 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/master-0/data/instance:
uuid: "4053a2fb000a4087b777d47afba15acb"
format_stamp: "Formatted at 2026-11-19 09:04:12 on dist-test-slave-1fcg"
I20261119 09:04:12.359248 538 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/master-0/wal/instance:
uuid: "4053a2fb000a4087b777d47afba15acb"
format_stamp: "Formatted at 2026-11-19 09:04:12 on dist-test-slave-1fcg"
I20261119 09:04:12.362754 538 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261119 09:04:12.365315 552 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:12.366462 538 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261119 09:04:12.366609 538 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/master-0/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/master-0/wal
uuid: "4053a2fb000a4087b777d47afba15acb"
format_stamp: "Formatted at 2026-11-19 09:04:12 on dist-test-slave-1fcg"
I20261119 09:04:12.366701 538 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:12.383414 538 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:12.383993 538 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:12.384145 538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:12.391499 538 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.126:39461
I20261119 09:04:12.391520 604 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.126:39461 every 8 connection(s)
I20261119 09:04:12.392664 538 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/master-0/data/info.pb
I20261119 09:04:12.395907 605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:12.400980 605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb: Bootstrap starting.
I20261119 09:04:12.402788 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 538
I20261119 09:04:12.402910 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/master-0/wal/instance
I20261119 09:04:12.403852 605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:12.404678 605 log.cc:826] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:12.406536 605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb: No bootstrap required, opened a new log
I20261119 09:04:12.409212 605 raft_consensus.cc:360] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4053a2fb000a4087b777d47afba15acb" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 39461 } }
I20261119 09:04:12.409392 605 raft_consensus.cc:386] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:04:12.409441 605 raft_consensus.cc:748] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4053a2fb000a4087b777d47afba15acb, State: Initialized, Role: FOLLOWER
I20261119 09:04:12.409830 605 consensus_queue.cc:261] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4053a2fb000a4087b777d47afba15acb" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 39461 } }
I20261119 09:04:12.409966 605 raft_consensus.cc:400] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261119 09:04:12.410022 605 raft_consensus.cc:499] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261119 09:04:12.410108 605 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:12.410856 605 raft_consensus.cc:521] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4053a2fb000a4087b777d47afba15acb" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 39461 } }
I20261119 09:04:12.411216 605 leader_election.cc:304] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4053a2fb000a4087b777d47afba15acb; no voters:
I20261119 09:04:12.411497 605 leader_election.cc:290] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb [CANDIDATE]: Term 1 election: Requested vote from peers
I20261119 09:04:12.411713 610 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb [term 1 FOLLOWER]: Leader election won for term 1
I20261119 09:04:12.411921 610 raft_consensus.cc:705] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb [term 1 LEADER]: Becoming Leader. State: Replica: 4053a2fb000a4087b777d47afba15acb, State: Running, Role: LEADER
I20261119 09:04:12.412197 610 consensus_queue.cc:238] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4053a2fb000a4087b777d47afba15acb" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 39461 } }
I20261119 09:04:12.412530 605 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb [sys.catalog]: configured and running, proceeding with master startup.
I20261119 09:04:12.413671 612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4053a2fb000a4087b777d47afba15acb. Latest consensus state: current_term: 1 leader_uuid: "4053a2fb000a4087b777d47afba15acb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4053a2fb000a4087b777d47afba15acb" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 39461 } } }
I20261119 09:04:12.413800 612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb [sys.catalog]: This master's current role is: LEADER
I20261119 09:04:12.413681 611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4053a2fb000a4087b777d47afba15acb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4053a2fb000a4087b777d47afba15acb" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 39461 } } }
I20261119 09:04:12.413895 611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb [sys.catalog]: This master's current role is: LEADER
I20261119 09:04:12.414835 619 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261119 09:04:12.417726 619 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261119 09:04:12.423231 619 catalog_manager.cc:1398] Generated new cluster ID: 2f809b4cb6ed4b80be8695d829370ace
I20261119 09:04:12.423303 619 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261119 09:04:12.448504 619 catalog_manager.cc:1421] Generated new certificate authority record
I20261119 09:04:12.449450 619 catalog_manager.cc:1555] Loading token signing keys...
I20261119 09:04:12.466097 619 catalog_manager.cc:6237] T 00000000000000000000000000000000 P 4053a2fb000a4087b777d47afba15acb: Generated new TSK 0
I20261119 09:04:12.466854 619 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261119 09:04:12.471655 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.65:0
--local_ip_for_outbound_sockets=127.30.252.65
--webserver_interface=127.30.252.65
--webserver_port=0
--tserver_master_addrs=127.30.252.126:39461
--builtin_ntp_servers=127.30.252.84:44289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--raft_prepare_replacement_before_eviction=true with env {}
W20261119 09:04:12.585732 629 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20261119 09:04:12.585927 629 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:12.585948 629 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:12.586001 629 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:12.589740 629 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:12.589882 629 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.65
I20261119 09:04:12.594198 629 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:44289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.252.65
--webserver_port=0
--tserver_master_addrs=127.30.252.126:39461
--never_fsync=true
--heap_profile_path=/tmp/kudu.629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:12.595073 629 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:12.595839 629 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:12.601785 637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:12.601904 634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:12.601913 635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:12.602152 629 server_base.cc:1061] running on GCE node
I20261119 09:04:12.602536 629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:12.603114 629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:12.604271 629 hybrid_clock.cc:648] HybridClock initialized: now 1795079052604237 us; error 48 us; skew 500 ppm
I20261119 09:04:12.606081 629 webserver.cc:577] Webserver started at http://127.30.252.65:40179/ using document root <none> and password file <none>
I20261119 09:04:12.606666 629 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:12.606720 629 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:12.607028 629 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:12.608420 629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-0/data/instance:
uuid: "9aa97314764e4bbbb163df9268fb32d8"
format_stamp: "Formatted at 2026-11-19 09:04:12 on dist-test-slave-1fcg"
I20261119 09:04:12.608829 629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-0/wal/instance:
uuid: "9aa97314764e4bbbb163df9268fb32d8"
format_stamp: "Formatted at 2026-11-19 09:04:12 on dist-test-slave-1fcg"
I20261119 09:04:12.612056 629 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20261119 09:04:12.614516 643 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:12.615710 629 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261119 09:04:12.615823 629 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-0/wal
uuid: "9aa97314764e4bbbb163df9268fb32d8"
format_stamp: "Formatted at 2026-11-19 09:04:12 on dist-test-slave-1fcg"
I20261119 09:04:12.615921 629 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:12.625617 629 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:12.626063 629 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:12.626201 629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:12.626672 629 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:12.627632 629 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:12.627677 629 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:12.627729 629 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:12.627758 629 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:12.637254 629 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.65:42771
I20261119 09:04:12.637416 756 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.65:42771 every 8 connection(s)
I20261119 09:04:12.638284 629 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-0/data/info.pb
I20261119 09:04:12.647472 757 heartbeater.cc:345] Connected to a master server at 127.30.252.126:39461
I20261119 09:04:12.647719 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 629
I20261119 09:04:12.647782 757 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:12.647827 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-0/wal/instance
I20261119 09:04:12.648388 757 heartbeater.cc:508] Master 127.30.252.126:39461 requested a full tablet report, sending...
I20261119 09:04:12.650177 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.66:0
--local_ip_for_outbound_sockets=127.30.252.66
--webserver_interface=127.30.252.66
--webserver_port=0
--tserver_master_addrs=127.30.252.126:39461
--builtin_ntp_servers=127.30.252.84:44289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--raft_prepare_replacement_before_eviction=true with env {}
I20261119 09:04:12.650169 569 ts_manager.cc:210] Registered new tserver with Master: 9aa97314764e4bbbb163df9268fb32d8 (127.30.252.65:42771)
I20261119 09:04:12.651618 569 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.65:35007
W20261119 09:04:12.759972 760 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20261119 09:04:12.760159 760 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:12.760180 760 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:12.760250 760 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:12.764003 760 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:12.764127 760 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.66
I20261119 09:04:12.768386 760 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:44289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.252.66
--webserver_port=0
--tserver_master_addrs=127.30.252.126:39461
--never_fsync=true
--heap_profile_path=/tmp/kudu.760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:12.769248 760 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:12.769961 760 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:12.776971 766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:12.776966 768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:12.776971 765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:12.777382 760 server_base.cc:1061] running on GCE node
I20261119 09:04:12.777812 760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:12.778523 760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:12.779649 760 hybrid_clock.cc:648] HybridClock initialized: now 1795079052779641 us; error 32 us; skew 500 ppm
I20261119 09:04:12.781766 760 webserver.cc:577] Webserver started at http://127.30.252.66:45087/ using document root <none> and password file <none>
I20261119 09:04:12.782505 760 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:12.782574 760 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:12.782817 760 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:12.784471 760 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-1/data/instance:
uuid: "9e3b24ac00b94322bafcbb19336fbd64"
format_stamp: "Formatted at 2026-11-19 09:04:12 on dist-test-slave-1fcg"
I20261119 09:04:12.784878 760 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-1/wal/instance:
uuid: "9e3b24ac00b94322bafcbb19336fbd64"
format_stamp: "Formatted at 2026-11-19 09:04:12 on dist-test-slave-1fcg"
I20261119 09:04:12.788475 760 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20261119 09:04:12.791559 774 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:12.792811 760 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20261119 09:04:12.792943 760 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-1/wal
uuid: "9e3b24ac00b94322bafcbb19336fbd64"
format_stamp: "Formatted at 2026-11-19 09:04:12 on dist-test-slave-1fcg"
I20261119 09:04:12.793049 760 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:12.806990 760 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:12.807479 760 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:12.807644 760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:12.808128 760 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:12.809077 760 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:12.809123 760 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:12.809165 760 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:12.809180 760 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:12.817763 760 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.66:38511
I20261119 09:04:12.817823 887 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.66:38511 every 8 connection(s)
I20261119 09:04:12.818846 760 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-1/data/info.pb
I20261119 09:04:12.826747 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 760
I20261119 09:04:12.826853 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-1/wal/instance
I20261119 09:04:12.827446 888 heartbeater.cc:345] Connected to a master server at 127.30.252.126:39461
I20261119 09:04:12.827708 888 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:12.828516 888 heartbeater.cc:508] Master 127.30.252.126:39461 requested a full tablet report, sending...
I20261119 09:04:12.829195 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.67:0
--local_ip_for_outbound_sockets=127.30.252.67
--webserver_interface=127.30.252.67
--webserver_port=0
--tserver_master_addrs=127.30.252.126:39461
--builtin_ntp_servers=127.30.252.84:44289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--raft_prepare_replacement_before_eviction=true with env {}
I20261119 09:04:12.829615 569 ts_manager.cc:210] Registered new tserver with Master: 9e3b24ac00b94322bafcbb19336fbd64 (127.30.252.66:38511)
I20261119 09:04:12.830178 569 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.66:44141
W20261119 09:04:12.933319 891 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20261119 09:04:12.933494 891 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:12.933516 891 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:12.933555 891 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:12.937228 891 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:12.937449 891 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.67
I20261119 09:04:12.941540 891 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:44289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.252.67
--webserver_port=0
--tserver_master_addrs=127.30.252.126:39461
--never_fsync=true
--heap_profile_path=/tmp/kudu.891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:12.942288 891 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:12.942978 891 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:12.948827 896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:12.948880 899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:12.948840 897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:12.949008 891 server_base.cc:1061] running on GCE node
I20261119 09:04:12.949426 891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:12.949926 891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:12.951129 891 hybrid_clock.cc:648] HybridClock initialized: now 1795079052951065 us; error 75 us; skew 500 ppm
I20261119 09:04:12.952898 891 webserver.cc:577] Webserver started at http://127.30.252.67:41945/ using document root <none> and password file <none>
I20261119 09:04:12.953430 891 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:12.953477 891 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:12.953655 891 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:12.954870 891 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-2/data/instance:
uuid: "0d13d70a226f49d7a2158376fdbd4670"
format_stamp: "Formatted at 2026-11-19 09:04:12 on dist-test-slave-1fcg"
I20261119 09:04:12.955250 891 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-2/wal/instance:
uuid: "0d13d70a226f49d7a2158376fdbd4670"
format_stamp: "Formatted at 2026-11-19 09:04:12 on dist-test-slave-1fcg"
I20261119 09:04:12.958452 891 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261119 09:04:12.960804 905 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:12.961805 891 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261119 09:04:12.961915 891 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-2/wal
uuid: "0d13d70a226f49d7a2158376fdbd4670"
format_stamp: "Formatted at 2026-11-19 09:04:12 on dist-test-slave-1fcg"
I20261119 09:04:12.962009 891 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:12.980579 891 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:12.981051 891 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:12.981185 891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:12.981686 891 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:12.982659 891 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:12.982705 891 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:12.982745 891 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:12.982771 891 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:12.992062 891 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.67:44033
I20261119 09:04:12.992136 1018 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.67:44033 every 8 connection(s)
I20261119 09:04:12.993080 891 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-2/data/info.pb
I20261119 09:04:12.995113 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 891
I20261119 09:04:12.995213 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-2/wal/instance
I20261119 09:04:12.997601 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.68:0
--local_ip_for_outbound_sockets=127.30.252.68
--webserver_interface=127.30.252.68
--webserver_port=0
--tserver_master_addrs=127.30.252.126:39461
--builtin_ntp_servers=127.30.252.84:44289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--raft_prepare_replacement_before_eviction=true with env {}
I20261119 09:04:13.003026 1019 heartbeater.cc:345] Connected to a master server at 127.30.252.126:39461
I20261119 09:04:13.003443 1019 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:13.004290 1019 heartbeater.cc:508] Master 127.30.252.126:39461 requested a full tablet report, sending...
I20261119 09:04:13.005767 569 ts_manager.cc:210] Registered new tserver with Master: 0d13d70a226f49d7a2158376fdbd4670 (127.30.252.67:44033)
I20261119 09:04:13.006652 569 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.67:35975
W20261119 09:04:13.107498 1022 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20261119 09:04:13.107758 1022 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:13.107791 1022 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:13.107851 1022 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:13.112115 1022 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:13.112275 1022 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.68
I20261119 09:04:13.117348 1022 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:44289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.30.252.68
--webserver_port=0
--tserver_master_addrs=127.30.252.126:39461
--never_fsync=true
--heap_profile_path=/tmp/kudu.1022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.68
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:13.118764 1022 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:13.119540 1022 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:13.125768 1027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:13.125766 1030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:13.125886 1028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:13.125972 1022 server_base.cc:1061] running on GCE node
I20261119 09:04:13.126405 1022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:13.126967 1022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:13.128137 1022 hybrid_clock.cc:648] HybridClock initialized: now 1795079053128109 us; error 38 us; skew 500 ppm
I20261119 09:04:13.130004 1022 webserver.cc:577] Webserver started at http://127.30.252.68:45909/ using document root <none> and password file <none>
I20261119 09:04:13.130585 1022 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:13.130635 1022 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:13.130828 1022 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:13.132333 1022 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-3/data/instance:
uuid: "947ea54f74a34522acff532e3d3a4a39"
format_stamp: "Formatted at 2026-11-19 09:04:13 on dist-test-slave-1fcg"
I20261119 09:04:13.132735 1022 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-3/wal/instance:
uuid: "947ea54f74a34522acff532e3d3a4a39"
format_stamp: "Formatted at 2026-11-19 09:04:13 on dist-test-slave-1fcg"
I20261119 09:04:13.136840 1022 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20261119 09:04:13.139392 1037 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:13.140578 1022 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261119 09:04:13.140673 1022 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-3/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-3/wal
uuid: "947ea54f74a34522acff532e3d3a4a39"
format_stamp: "Formatted at 2026-11-19 09:04:13 on dist-test-slave-1fcg"
I20261119 09:04:13.140805 1022 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:13.168296 1022 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:13.168807 1022 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:13.168944 1022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:13.169467 1022 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:13.170555 1022 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:13.170616 1022 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:13.170668 1022 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:13.170703 1022 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:13.181361 1022 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.68:42755
I20261119 09:04:13.181372 1150 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.68:42755 every 8 connection(s)
I20261119 09:04:13.182382 1022 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-3/data/info.pb
I20261119 09:04:13.184834 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 1022
I20261119 09:04:13.184937 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795079038760673-31729-0/minicluster-data/ts-3/wal/instance
I20261119 09:04:13.191124 1151 heartbeater.cc:345] Connected to a master server at 127.30.252.126:39461
I20261119 09:04:13.191449 1151 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:13.192309 1151 heartbeater.cc:508] Master 127.30.252.126:39461 requested a full tablet report, sending...
I20261119 09:04:13.193681 569 ts_manager.cc:210] Registered new tserver with Master: 947ea54f74a34522acff532e3d3a4a39 (127.30.252.68:42755)
I20261119 09:04:13.194254 569 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.68:58513
I20261119 09:04:13.199594 31729 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261119 09:04:13.219885 569 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50788:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261119 09:04:13.238488 822 tablet_service.cc:1495] Processing CreateTablet for tablet ab6a970ff9b1431b90fcf9c4e72fe6f9 (DEFAULT_TABLE table=test-workload [id=847fa96d794b41aca6ca70d9309846d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20261119 09:04:13.238740 691 tablet_service.cc:1495] Processing CreateTablet for tablet ab6a970ff9b1431b90fcf9c4e72fe6f9 (DEFAULT_TABLE table=test-workload [id=847fa96d794b41aca6ca70d9309846d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20261119 09:04:13.238740 953 tablet_service.cc:1495] Processing CreateTablet for tablet ab6a970ff9b1431b90fcf9c4e72fe6f9 (DEFAULT_TABLE table=test-workload [id=847fa96d794b41aca6ca70d9309846d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20261119 09:04:13.239452 822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab6a970ff9b1431b90fcf9c4e72fe6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:13.239914 691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab6a970ff9b1431b90fcf9c4e72fe6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:13.239921 953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab6a970ff9b1431b90fcf9c4e72fe6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:13.245018 1173 tablet_bootstrap.cc:492] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64: Bootstrap starting.
I20261119 09:04:13.246186 1174 tablet_bootstrap.cc:492] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8: Bootstrap starting.
I20261119 09:04:13.247716 1173 tablet_bootstrap.cc:654] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:13.248051 1175 tablet_bootstrap.cc:492] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670: Bootstrap starting.
I20261119 09:04:13.248514 1174 tablet_bootstrap.cc:654] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:13.248672 1173 log.cc:826] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:13.249374 1174 log.cc:826] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:13.250296 1175 tablet_bootstrap.cc:654] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:13.250485 1173 tablet_bootstrap.cc:492] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64: No bootstrap required, opened a new log
I20261119 09:04:13.250712 1173 ts_tablet_manager.cc:1449] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20261119 09:04:13.251011 1174 tablet_bootstrap.cc:492] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8: No bootstrap required, opened a new log
I20261119 09:04:13.251058 1175 log.cc:826] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:13.251209 1174 ts_tablet_manager.cc:1449] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20261119 09:04:13.252259 1175 tablet_bootstrap.cc:492] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670: No bootstrap required, opened a new log
I20261119 09:04:13.252444 1175 ts_tablet_manager.cc:1449] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20261119 09:04:13.253980 1173 raft_consensus.cc:360] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } }
I20261119 09:04:13.254222 1173 raft_consensus.cc:748] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3b24ac00b94322bafcbb19336fbd64, State: Initialized, Role: FOLLOWER
I20261119 09:04:13.254132 1174 raft_consensus.cc:360] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } }
I20261119 09:04:13.254288 1174 raft_consensus.cc:748] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa97314764e4bbbb163df9268fb32d8, State: Initialized, Role: FOLLOWER
I20261119 09:04:13.254595 1174 consensus_queue.cc:261] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } }
I20261119 09:04:13.254595 1173 consensus_queue.cc:261] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } }
I20261119 09:04:13.254987 1175 raft_consensus.cc:360] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } }
I20261119 09:04:13.255198 1175 raft_consensus.cc:748] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d13d70a226f49d7a2158376fdbd4670, State: Initialized, Role: FOLLOWER
I20261119 09:04:13.255429 1173 ts_tablet_manager.cc:1480] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20261119 09:04:13.255429 1174 ts_tablet_manager.cc:1480] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20261119 09:04:13.255519 1175 consensus_queue.cc:261] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } }
I20261119 09:04:13.255827 888 heartbeater.cc:500] Master 127.30.252.126:39461 was elected leader, sending a full tablet report...
I20261119 09:04:13.255827 757 heartbeater.cc:500] Master 127.30.252.126:39461 was elected leader, sending a full tablet report...
I20261119 09:04:13.256299 1175 ts_tablet_manager.cc:1480] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20261119 09:04:13.257494 1019 heartbeater.cc:500] Master 127.30.252.126:39461 was elected leader, sending a full tablet report...
I20261119 09:04:13.286649 711 tablet_service.cc:2028] Received Run Leader Election RPC: tablet_id: "ab6a970ff9b1431b90fcf9c4e72fe6f9"
dest_uuid: "9aa97314764e4bbbb163df9268fb32d8"
from {username='slave'} at 127.0.0.1:60346
I20261119 09:04:13.286875 711 raft_consensus.cc:499] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20261119 09:04:13.286966 711 raft_consensus.cc:3096] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:13.288556 711 raft_consensus.cc:521] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } }
I20261119 09:04:13.289988 711 leader_election.cc:290] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [CANDIDATE]: Term 1 election: Requested vote from peers 0d13d70a226f49d7a2158376fdbd4670 (127.30.252.67:44033), 9e3b24ac00b94322bafcbb19336fbd64 (127.30.252.66:38511)
I20261119 09:04:13.294644 973 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ab6a970ff9b1431b90fcf9c4e72fe6f9" candidate_uuid: "9aa97314764e4bbbb163df9268fb32d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0d13d70a226f49d7a2158376fdbd4670"
I20261119 09:04:13.294898 973 raft_consensus.cc:3096] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:13.295154 842 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "ab6a970ff9b1431b90fcf9c4e72fe6f9" candidate_uuid: "9aa97314764e4bbbb163df9268fb32d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9e3b24ac00b94322bafcbb19336fbd64"
I20261119 09:04:13.295351 842 raft_consensus.cc:3096] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64 [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:13.296512 973 raft_consensus.cc:2490] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9aa97314764e4bbbb163df9268fb32d8 in term 1.
I20261119 09:04:13.296657 842 raft_consensus.cc:2490] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9aa97314764e4bbbb163df9268fb32d8 in term 1.
I20261119 09:04:13.297134 646 leader_election.cc:304] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9aa97314764e4bbbb163df9268fb32d8, 9e3b24ac00b94322bafcbb19336fbd64; no voters:
I20261119 09:04:13.298084 1180 raft_consensus.cc:2826] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [term 1 FOLLOWER]: Leader election won for term 1
I20261119 09:04:13.298328 1180 raft_consensus.cc:705] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [term 1 LEADER]: Becoming Leader. State: Replica: 9aa97314764e4bbbb163df9268fb32d8, State: Running, Role: LEADER
I20261119 09:04:13.298723 1180 consensus_queue.cc:238] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } }
I20261119 09:04:13.302604 562 catalog_manager.cc:5856] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9aa97314764e4bbbb163df9268fb32d8 (127.30.252.65). New cstate: current_term: 1 leader_uuid: "9aa97314764e4bbbb163df9268fb32d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } health_report { overall_health: HEALTHY } } }
I20261119 09:04:13.304458 1180 consensus_queue.cc:1049] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261119 09:04:13.307471 1189 consensus_queue.cc:1049] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261119 09:04:13.363979 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 538
I20261119 09:04:13.372763 31729 raft_config_change-itest.cc:349] submitting config change with changes:
I20261119 09:04:13.372843 31729 raft_config_change-itest.cc:351] type: ADD_PEER peer { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } }
I20261119 09:04:13.376689 711 consensus_queue.cc:238] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } }
I20261119 09:04:13.378827 973 raft_consensus.cc:1285] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 1 FOLLOWER]: Refusing update from remote peer 9aa97314764e4bbbb163df9268fb32d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20261119 09:04:13.379288 1189 consensus_queue.cc:1049] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20261119 09:04:13.380304 842 raft_consensus.cc:1285] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64 [term 1 FOLLOWER]: Refusing update from remote peer 9aa97314764e4bbbb163df9268fb32d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20261119 09:04:13.380842 1189 consensus_queue.cc:1049] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20261119 09:04:13.382332 1189 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index -1 to 10, VOTER 947ea54f74a34522acff532e3d3a4a39 (127.30.252.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } }
I20261119 09:04:13.382764 842 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index -1 to 10, VOTER 947ea54f74a34522acff532e3d3a4a39 (127.30.252.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } }
W20261119 09:04:13.385515 888 heartbeater.cc:648] Failed to heartbeat to 127.30.252.126:39461 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.252.126:39461: connect: Connection refused (error 111)
I20261119 09:04:13.385546 31729 raft_config_change-itest.cc:349] submitting config change with changes:
I20261119 09:04:13.385586 31729 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } }
I20261119 09:04:13.385663 31729 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: true replace: false } }
W20261119 09:04:13.386057 644 consensus_peers.cc:597] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 -> Peer 947ea54f74a34522acff532e3d3a4a39 (127.30.252.68:42755): Couldn't send request to peer 947ea54f74a34522acff532e3d3a4a39. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab6a970ff9b1431b90fcf9c4e72fe6f9. This is attempt 1: this message will repeat every 5th retry.
W20261119 09:04:13.386332 757 heartbeater.cc:648] Failed to heartbeat to 127.30.252.126:39461 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.252.126:39461: connect: Connection refused (error 111)
I20261119 09:04:13.386646 711 consensus_queue.cc:238] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: true replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } }
I20261119 09:04:13.386664 973 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index -1 to 10, VOTER 947ea54f74a34522acff532e3d3a4a39 (127.30.252.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } }
W20261119 09:04:13.388130 644 consensus_peers.cc:597] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 -> Peer 947ea54f74a34522acff532e3d3a4a39 (127.30.252.68:42755): Couldn't send request to peer 947ea54f74a34522acff532e3d3a4a39. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab6a970ff9b1431b90fcf9c4e72fe6f9. This is attempt 1: this message will repeat every 5th retry.
I20261119 09:04:13.388413 972 raft_consensus.cc:1285] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 1 FOLLOWER]: Refusing update from remote peer 9aa97314764e4bbbb163df9268fb32d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20261119 09:04:13.388808 842 raft_consensus.cc:1285] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64 [term 1 FOLLOWER]: Refusing update from remote peer 9aa97314764e4bbbb163df9268fb32d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20261119 09:04:13.389027 1180 consensus_queue.cc:1049] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: true replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20261119 09:04:13.389269 1019 heartbeater.cc:648] Failed to heartbeat to 127.30.252.126:39461 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.252.126:39461: connect: Connection refused (error 111)
I20261119 09:04:13.389283 1180 consensus_queue.cc:1049] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20261119 09:04:13.390377 1180 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9e3b24ac00b94322bafcbb19336fbd64 (127.30.252.66) changed from VOTER to NON_VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: true replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } }
I20261119 09:04:13.391165 842 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64 [term 1 LEARNER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9e3b24ac00b94322bafcbb19336fbd64 (127.30.252.66) changed from VOTER to NON_VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: true replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } }
I20261119 09:04:13.392284 972 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9e3b24ac00b94322bafcbb19336fbd64 (127.30.252.66) changed from VOTER to NON_VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: true replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } }
I20261119 09:04:13.392799 31729 raft_config_change-itest.cc:349] submitting config change with changes:
I20261119 09:04:13.392844 31729 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: true } }
I20261119 09:04:13.393677 711 consensus_queue.cc:238] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: true replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: true } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } }
I20261119 09:04:13.394893 842 raft_consensus.cc:1285] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64 [term 1 LEARNER]: Refusing update from remote peer 9aa97314764e4bbbb163df9268fb32d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20261119 09:04:13.394937 972 raft_consensus.cc:1285] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 1 FOLLOWER]: Refusing update from remote peer 9aa97314764e4bbbb163df9268fb32d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20261119 09:04:13.395133 644 consensus_peers.cc:597] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 -> Peer 947ea54f74a34522acff532e3d3a4a39 (127.30.252.68:42755): Couldn't send request to peer 947ea54f74a34522acff532e3d3a4a39. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab6a970ff9b1431b90fcf9c4e72fe6f9. This is attempt 1: this message will repeat every 5th retry.
I20261119 09:04:13.395217 1191 consensus_queue.cc:1049] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20261119 09:04:13.395314 1192 consensus_queue.cc:1049] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: true replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20261119 09:04:13.396801 1192 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: true replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: true } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } }
I20261119 09:04:13.397176 972 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: true replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: true } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } }
I20261119 09:04:13.399443 842 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64 [term 1 LEARNER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: true replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: true } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } }
I20261119 09:04:13.400435 31729 raft_config_change-itest.cc:349] submitting config change with changes:
I20261119 09:04:13.400483 31729 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } }
I20261119 09:04:13.400563 31729 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: NON_VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: false replace: false } }
I20261119 09:04:13.401705 31729 raft_config_change-itest.cc:349] submitting config change with changes:
I20261119 09:04:13.401753 31729 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: NON_VOTER last_known_addr { host: "127.30.252.65" port: 42771 } attrs { promote: false replace: false } }
I20261119 09:04:13.403308 31729 raft_config_change-itest.cc:349] submitting config change with changes:
I20261119 09:04:13.403352 31729 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: true replace: false } }
I20261119 09:04:13.404868 31729 raft_config_change-itest.cc:349] submitting config change with changes:
I20261119 09:04:13.404911 31729 raft_config_change-itest.cc:351] type: REMOVE_PEER peer { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: UNKNOWN_MEMBER_TYPE last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } }
I20261119 09:04:13.404980 31729 raft_config_change-itest.cc:351] type: REMOVE_PEER peer { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: UNKNOWN_MEMBER_TYPE last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: false replace: false } }
I20261119 09:04:13.405802 711 consensus_queue.cc:238] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } }
W20261119 09:04:13.407028 644 consensus_peers.cc:597] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 -> Peer 947ea54f74a34522acff532e3d3a4a39 (127.30.252.68:42755): Couldn't send request to peer 947ea54f74a34522acff532e3d3a4a39. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab6a970ff9b1431b90fcf9c4e72fe6f9. This is attempt 1: this message will repeat every 5th retry.
I20261119 09:04:13.886612 1214 ts_tablet_manager.cc:979] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39: Initiating tablet copy from peer 9aa97314764e4bbbb163df9268fb32d8 (127.30.252.65:42771)
I20261119 09:04:13.887117 1214 tablet_copy_client.cc:323] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39: tablet copy: Beginning tablet copy session from remote peer at address 127.30.252.65:42771
I20261119 09:04:13.890794 731 tablet_copy_service.cc:140] P 9aa97314764e4bbbb163df9268fb32d8: Received BeginTabletCopySession request for tablet ab6a970ff9b1431b90fcf9c4e72fe6f9 from peer 947ea54f74a34522acff532e3d3a4a39 ({username='slave'} at 127.30.252.68:49127)
I20261119 09:04:13.890947 731 tablet_copy_service.cc:161] P 9aa97314764e4bbbb163df9268fb32d8: Beginning new tablet copy session on tablet ab6a970ff9b1431b90fcf9c4e72fe6f9 from peer 947ea54f74a34522acff532e3d3a4a39 at {username='slave'} at 127.30.252.68:49127: session id = 947ea54f74a34522acff532e3d3a4a39-ab6a970ff9b1431b90fcf9c4e72fe6f9
I20261119 09:04:13.892184 731 tablet_copy_source_session.cc:215] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8: Tablet Copy: opened 0 blocks and 1 log segments
I20261119 09:04:13.895831 1214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab6a970ff9b1431b90fcf9c4e72fe6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:13.900893 1214 tablet_copy_client.cc:806] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39: tablet copy: Starting download of 0 data blocks...
I20261119 09:04:13.901088 1214 tablet_copy_client.cc:670] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39: tablet copy: Starting download of 1 WAL segments...
I20261119 09:04:13.903002 1214 tablet_copy_client.cc:538] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261119 09:04:13.904812 1214 tablet_bootstrap.cc:492] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39: Bootstrap starting.
I20261119 09:04:13.910352 1214 log.cc:826] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:13.927783 1214 tablet_bootstrap.cc:492] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=12 ignored=0} inserts{seen=400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20261119 09:04:13.928169 1214 tablet_bootstrap.cc:492] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39: Bootstrap complete.
I20261119 09:04:13.928505 1214 ts_tablet_manager.cc:1449] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39: Time spent bootstrapping tablet: real 0.024s user 0.018s sys 0.005s
I20261119 09:04:13.930562 1214 raft_consensus.cc:360] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: true replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: true } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } }
I20261119 09:04:13.931041 1214 raft_consensus.cc:748] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 947ea54f74a34522acff532e3d3a4a39, State: Initialized, Role: FOLLOWER
I20261119 09:04:13.931335 1214 consensus_queue.cc:261] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } }
I20261119 09:04:13.932024 1151 heartbeater.cc:500] Master 127.30.252.126:39461 was elected leader, sending a full tablet report...
I20261119 09:04:13.932113 1214 ts_tablet_manager.cc:1480] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20261119 09:04:13.932971 731 tablet_copy_service.cc:342] P 9aa97314764e4bbbb163df9268fb32d8: Request end of tablet copy session 947ea54f74a34522acff532e3d3a4a39-ab6a970ff9b1431b90fcf9c4e72fe6f9 received from {username='slave'} at 127.30.252.68:49127
I20261119 09:04:13.933120 731 tablet_copy_service.cc:434] P 9aa97314764e4bbbb163df9268fb32d8: ending tablet copy session 947ea54f74a34522acff532e3d3a4a39-ab6a970ff9b1431b90fcf9c4e72fe6f9 on tablet ab6a970ff9b1431b90fcf9c4e72fe6f9 with peer 947ea54f74a34522acff532e3d3a4a39
W20261119 09:04:13.933737 1151 heartbeater.cc:648] Failed to heartbeat to 127.30.252.126:39461 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.252.126:39461: connect: Connection refused (error 111)
I20261119 09:04:14.432179 1105 raft_consensus.cc:1227] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20261119 09:04:14.433348 1225 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0d13d70a226f49d7a2158376fdbd4670 (127.30.252.67) evicted, NON_VOTER 9e3b24ac00b94322bafcbb19336fbd64 (127.30.252.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } }
I20261119 09:04:14.433907 1105 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0d13d70a226f49d7a2158376fdbd4670 (127.30.252.67) evicted, NON_VOTER 9e3b24ac00b94322bafcbb19336fbd64 (127.30.252.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } }
I20261119 09:04:14.434580 1225 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:60346 (request call id 11) took 1029 ms. Trace:
I20261119 09:04:14.434919 1225 rpcz_store.cc:276] 1119 09:04:13.405377 (+ 0us) service_pool.cc:167] Inserting onto call queue
1119 09:04:13.405465 (+ 88us) service_pool.cc:224] Handling call
1119 09:04:13.406133 (+ 668us) log.cc:844] Serialized 454 byte log entry
1119 09:04:14.434563 (+1028430us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":37,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":107}
I20261119 09:04:14.436688 31729 raft_config_change-itest.cc:349] submitting config change with changes:
I20261119 09:04:14.436733 31729 raft_config_change-itest.cc:351] type: ADD_PEER peer { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } }
I20261119 09:04:14.436801 31729 raft_config_change-itest.cc:351] type: ADD_PEER peer { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: false replace: false } }
I20261119 09:04:14.437750 31729 raft_config_change-itest.cc:349] submitting config change with changes:
I20261119 09:04:14.437788 31729 raft_config_change-itest.cc:351] type: ADD_PEER peer { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } }
I20261119 09:04:14.438541 711 consensus_queue.cc:238] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } }
I20261119 09:04:14.439669 1105 raft_consensus.cc:1285] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39 [term 1 FOLLOWER]: Refusing update from remote peer 9aa97314764e4bbbb163df9268fb32d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20261119 09:04:14.439694 842 raft_consensus.cc:1285] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64 [term 1 LEARNER]: Refusing update from remote peer 9aa97314764e4bbbb163df9268fb32d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20261119 09:04:14.440115 1223 consensus_queue.cc:1049] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20261119 09:04:14.440289 1225 consensus_queue.cc:1049] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20261119 09:04:14.441534 842 raft_consensus.cc:879] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: true replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: true } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } }.
Pending config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } }:
New config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } }
W20261119 09:04:14.441591 842 raft_consensus.cc:1590] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 14' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: true replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: true } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } }.
Pending config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } }:
New config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } }
I20261119 09:04:14.441977 842 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0d13d70a226f49d7a2158376fdbd4670 (127.30.252.67) evicted, NON_VOTER 9e3b24ac00b94322bafcbb19336fbd64 (127.30.252.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } }
I20261119 09:04:14.442014 1223 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 9e3b24ac00b94322bafcbb19336fbd64 (127.30.252.66) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } } }
I20261119 09:04:14.442482 1105 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 9e3b24ac00b94322bafcbb19336fbd64 (127.30.252.66) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } } }
I20261119 09:04:14.444612 842 raft_consensus.cc:1491] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20261119 09:04:14.444780 31729 raft_config_change-itest.cc:349] submitting config change with changes:
I20261119 09:04:14.444820 31729 raft_config_change-itest.cc:351] type: ADD_PEER peer { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: false replace: false } }
I20261119 09:04:14.444974 842 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 9e3b24ac00b94322bafcbb19336fbd64 (127.30.252.66) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } } }
I20261119 09:04:14.445492 711 consensus_queue.cc:238] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } } peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: false replace: false } }
I20261119 09:04:14.446734 972 raft_consensus.cc:1285] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 1 FOLLOWER]: Refusing update from remote peer 9aa97314764e4bbbb163df9268fb32d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20261119 09:04:14.446864 1105 raft_consensus.cc:1285] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39 [term 1 FOLLOWER]: Refusing update from remote peer 9aa97314764e4bbbb163df9268fb32d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20261119 09:04:14.447141 1225 consensus_queue.cc:1049] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: false replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.000s
I20261119 09:04:14.447371 1224 consensus_queue.cc:1049] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20261119 09:04:14.447537 842 raft_consensus.cc:1285] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64 [term 1 FOLLOWER]: Refusing update from remote peer 9aa97314764e4bbbb163df9268fb32d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20261119 09:04:14.447883 1224 consensus_queue.cc:1049] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20261119 09:04:14.448315 972 raft_consensus.cc:879] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: true replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: true } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } }.
Pending config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } }:
New config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } }
W20261119 09:04:14.448395 972 raft_consensus.cc:1590] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 14' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: true replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: NON_VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: true } } peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } }.
Pending config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } }:
New config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } }
I20261119 09:04:14.448813 972 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 0d13d70a226f49d7a2158376fdbd4670 (127.30.252.67) evicted, NON_VOTER 9e3b24ac00b94322bafcbb19336fbd64 (127.30.252.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } }
I20261119 09:04:14.450753 1224 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9aa97314764e4bbbb163df9268fb32d8 [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 0d13d70a226f49d7a2158376fdbd4670 (127.30.252.67) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } } peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: false replace: false } } }
I20261119 09:04:14.451047 1105 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 947ea54f74a34522acff532e3d3a4a39 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 0d13d70a226f49d7a2158376fdbd4670 (127.30.252.67) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } } peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: false replace: false } } }
I20261119 09:04:14.451175 972 raft_consensus.cc:1491] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20261119 09:04:14.451396 972 raft_consensus.cc:879] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } }.
Pending config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } }:
New config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } } peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: false replace: false } }
W20261119 09:04:14.451429 972 raft_consensus.cc:1590] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 15' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } }.
Pending config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } }:
New config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } } peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: false replace: false } }
I20261119 09:04:14.451596 972 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 9e3b24ac00b94322bafcbb19336fbd64 (127.30.252.66) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } } }
I20261119 09:04:14.452965 842 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 9e3b24ac00b94322bafcbb19336fbd64 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 0d13d70a226f49d7a2158376fdbd4670 (127.30.252.67) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } } peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: false replace: false } } }
I20261119 09:04:14.454036 31729 raft_config_change-itest.cc:349] submitting config change with changes:
I20261119 09:04:14.454084 31729 raft_config_change-itest.cc:351] type: REMOVE_PEER peer { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: UNKNOWN_MEMBER_TYPE last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } }
I20261119 09:04:14.454149 31729 raft_config_change-itest.cc:351] type: REMOVE_PEER peer { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: UNKNOWN_MEMBER_TYPE last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: false replace: false } }
I20261119 09:04:14.454716 972 raft_consensus.cc:1491] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20261119 09:04:14.455031 972 raft_consensus.cc:2983] T ab6a970ff9b1431b90fcf9c4e72fe6f9 P 0d13d70a226f49d7a2158376fdbd4670 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 0d13d70a226f49d7a2158376fdbd4670 (127.30.252.67) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9aa97314764e4bbbb163df9268fb32d8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 42771 } } peers { permanent_uuid: "947ea54f74a34522acff532e3d3a4a39" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42755 } attrs { promote: false replace: false } } peers { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } } peers { permanent_uuid: "0d13d70a226f49d7a2158376fdbd4670" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44033 } attrs { promote: false replace: false } } }
I20261119 09:04:14.455689 31729 raft_config_change-itest.cc:349] submitting config change with changes:
I20261119 09:04:14.455730 31729 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: false } }
I20261119 09:04:14.456872 31729 raft_config_change-itest.cc:349] submitting config change with changes:
I20261119 09:04:14.457846 31729 raft_config_change-itest.cc:349] submitting config change with changes:
I20261119 09:04:14.457885 31729 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: false replace: true } }
I20261119 09:04:14.457948 31729 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "9e3b24ac00b94322bafcbb19336fbd64" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 38511 } attrs { promote: true replace: false } }
I20261119 09:04:14.459319 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 629
I20261119 09:04:14.465528 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 760
I20261119 09:04:14.470759 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 891
I20261119 09:04:14.475792 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 1022
2026-11-19T09:04:14Z chronyd exiting
[ OK ] RaftConfigChangeITest.TestBulkChangeConfig (2278 ms)
[----------] 3 tests from RaftConfigChangeITest (15726 ms total)
[----------] 3 tests from SlowTabletBootstrapTest
[ RUN ] SlowTabletBootstrapTest.TestSlowBootstrap
I20261119 09:04:14.498320 31729 ts_itest-base.cc:115] Starting cluster with:
I20261119 09:04:14.498365 31729 ts_itest-base.cc:116] --------------
I20261119 09:04:14.498375 31729 ts_itest-base.cc:117] 4 tablet servers
I20261119 09:04:14.498391 31729 ts_itest-base.cc:118] 3 replicas per TS
I20261119 09:04:14.498400 31729 ts_itest-base.cc:119] --------------
2026-11-19T09:04:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-11-19T09:04:14Z Disabled control of system clock
I20261119 09:04:14.506451 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.252.126:41673
--webserver_interface=127.30.252.126
--webserver_port=0
--builtin_ntp_servers=127.30.252.84:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.252.126:41673 with env {}
W20261119 09:04:14.610579 1240 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:14.610765 1240 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:14.610805 1240 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:14.614454 1240 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261119 09:04:14.614535 1240 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:14.614565 1240 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261119 09:04:14.614584 1240 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261119 09:04:14.618768 1240 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.252.126:41673
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.252.126:41673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.252.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:14.619863 1240 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:14.620591 1240 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:14.626245 1248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:14.626346 1246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:14.626272 1245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:14.626678 1240 server_base.cc:1061] running on GCE node
I20261119 09:04:14.627081 1240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:14.627835 1240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:14.629010 1240 hybrid_clock.cc:648] HybridClock initialized: now 1795079054628991 us; error 32 us; skew 500 ppm
I20261119 09:04:14.630635 1240 webserver.cc:577] Webserver started at http://127.30.252.126:42083/ using document root <none> and password file <none>
I20261119 09:04:14.631222 1240 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:14.631268 1240 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:14.631474 1240 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:14.632787 1240 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e4448d060d6d47e48c765859fcfbdb46"
format_stamp: "Formatted at 2026-11-19 09:04:14 on dist-test-slave-1fcg"
I20261119 09:04:14.633210 1240 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e4448d060d6d47e48c765859fcfbdb46"
format_stamp: "Formatted at 2026-11-19 09:04:14 on dist-test-slave-1fcg"
I20261119 09:04:14.636656 1240 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20261119 09:04:14.639328 1254 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:14.640486 1240 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261119 09:04:14.640677 1240 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e4448d060d6d47e48c765859fcfbdb46"
format_stamp: "Formatted at 2026-11-19 09:04:14 on dist-test-slave-1fcg"
I20261119 09:04:14.640805 1240 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:14.665371 1240 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:14.665876 1240 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:14.666007 1240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:14.673097 1240 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.126:41673
I20261119 09:04:14.673166 1306 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.126:41673 every 8 connection(s)
I20261119 09:04:14.674160 1240 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20261119 09:04:14.677651 1307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:14.681844 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 1240
I20261119 09:04:14.681944 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal/instance
I20261119 09:04:14.683027 1307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46: Bootstrap starting.
I20261119 09:04:14.685801 1307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:14.686715 1307 log.cc:826] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:14.688409 1307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46: No bootstrap required, opened a new log
I20261119 09:04:14.691684 1307 raft_consensus.cc:360] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4448d060d6d47e48c765859fcfbdb46" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 41673 } }
I20261119 09:04:14.691874 1307 raft_consensus.cc:386] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:04:14.691921 1307 raft_consensus.cc:748] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4448d060d6d47e48c765859fcfbdb46, State: Initialized, Role: FOLLOWER
I20261119 09:04:14.692320 1307 consensus_queue.cc:261] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4448d060d6d47e48c765859fcfbdb46" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 41673 } }
I20261119 09:04:14.692487 1307 raft_consensus.cc:400] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261119 09:04:14.692548 1307 raft_consensus.cc:499] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261119 09:04:14.692631 1307 raft_consensus.cc:3096] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46 [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:14.693363 1307 raft_consensus.cc:521] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4448d060d6d47e48c765859fcfbdb46" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 41673 } }
I20261119 09:04:14.693728 1307 leader_election.cc:304] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4448d060d6d47e48c765859fcfbdb46; no voters:
I20261119 09:04:14.694000 1307 leader_election.cc:290] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261119 09:04:14.694123 1312 raft_consensus.cc:2826] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46 [term 1 FOLLOWER]: Leader election won for term 1
I20261119 09:04:14.694320 1312 raft_consensus.cc:705] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46 [term 1 LEADER]: Becoming Leader. State: Replica: e4448d060d6d47e48c765859fcfbdb46, State: Running, Role: LEADER
I20261119 09:04:14.694640 1312 consensus_queue.cc:238] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4448d060d6d47e48c765859fcfbdb46" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 41673 } }
I20261119 09:04:14.695009 1307 sys_catalog.cc:565] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46 [sys.catalog]: configured and running, proceeding with master startup.
I20261119 09:04:14.696133 1313 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4448d060d6d47e48c765859fcfbdb46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4448d060d6d47e48c765859fcfbdb46" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 41673 } } }
I20261119 09:04:14.696250 1313 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46 [sys.catalog]: This master's current role is: LEADER
I20261119 09:04:14.696604 1321 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261119 09:04:14.696625 1314 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4448d060d6d47e48c765859fcfbdb46. Latest consensus state: current_term: 1 leader_uuid: "e4448d060d6d47e48c765859fcfbdb46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4448d060d6d47e48c765859fcfbdb46" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 41673 } } }
I20261119 09:04:14.697438 1314 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46 [sys.catalog]: This master's current role is: LEADER
I20261119 09:04:14.700409 1321 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261119 09:04:14.705659 1321 catalog_manager.cc:1398] Generated new cluster ID: 07525d3f3aae4b08b0a7a00ddd8274cb
I20261119 09:04:14.705725 1321 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261119 09:04:14.728426 1321 catalog_manager.cc:1421] Generated new certificate authority record
I20261119 09:04:14.729555 1321 catalog_manager.cc:1555] Loading token signing keys...
I20261119 09:04:14.737151 1321 catalog_manager.cc:6237] T 00000000000000000000000000000000 P e4448d060d6d47e48c765859fcfbdb46: Generated new TSK 0
I20261119 09:04:14.738122 1321 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261119 09:04:14.754081 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.65:0
--local_ip_for_outbound_sockets=127.30.252.65
--webserver_interface=127.30.252.65
--webserver_port=0
--tserver_master_addrs=127.30.252.126:41673
--builtin_ntp_servers=127.30.252.84:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20261119 09:04:14.865537 1331 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261119 09:04:14.865779 1331 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:14.865823 1331 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:14.865899 1331 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:14.870065 1331 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261119 09:04:14.870147 1331 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261119 09:04:14.870203 1331 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:14.870303 1331 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261119 09:04:14.870347 1331 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.65
I20261119 09:04:14.874910 1331 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.252.65
--webserver_port=0
--tserver_master_addrs=127.30.252.126:41673
--never_fsync=true
--heap_profile_path=/tmp/kudu.1331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:14.875746 1331 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:14.876426 1331 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:14.882197 1339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:14.882190 1336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:14.882295 1331 server_base.cc:1061] running on GCE node
W20261119 09:04:14.882225 1337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:14.882705 1331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:14.883306 1331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:14.884467 1331 hybrid_clock.cc:648] HybridClock initialized: now 1795079054884449 us; error 35 us; skew 500 ppm
I20261119 09:04:14.886191 1331 webserver.cc:577] Webserver started at http://127.30.252.65:44197/ using document root <none> and password file <none>
I20261119 09:04:14.886766 1331 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:14.886823 1331 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:14.887032 1331 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:14.888401 1331 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c77ec8ad79fa4aeeabda528ce9b285ca"
format_stamp: "Formatted at 2026-11-19 09:04:14 on dist-test-slave-1fcg"
I20261119 09:04:14.888830 1331 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c77ec8ad79fa4aeeabda528ce9b285ca"
format_stamp: "Formatted at 2026-11-19 09:04:14 on dist-test-slave-1fcg"
I20261119 09:04:14.892071 1331 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20261119 09:04:14.894438 1345 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:14.895431 1331 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261119 09:04:14.895545 1331 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c77ec8ad79fa4aeeabda528ce9b285ca"
format_stamp: "Formatted at 2026-11-19 09:04:14 on dist-test-slave-1fcg"
I20261119 09:04:14.895663 1331 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:14.923317 1331 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:14.923825 1331 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:14.923974 1331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:14.924474 1331 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:14.925595 1331 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:14.925655 1331 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:14.925716 1331 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:14.925745 1331 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:14.934297 1331 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.65:44327
I20261119 09:04:14.934387 1458 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.65:44327 every 8 connection(s)
I20261119 09:04:14.935292 1331 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20261119 09:04:14.940524 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 1331
I20261119 09:04:14.940622 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20261119 09:04:14.943045 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.66:0
--local_ip_for_outbound_sockets=127.30.252.66
--webserver_interface=127.30.252.66
--webserver_port=0
--tserver_master_addrs=127.30.252.126:41673
--builtin_ntp_servers=127.30.252.84:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261119 09:04:14.943787 1459 heartbeater.cc:345] Connected to a master server at 127.30.252.126:41673
I20261119 09:04:14.944058 1459 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:14.944674 1459 heartbeater.cc:508] Master 127.30.252.126:41673 requested a full tablet report, sending...
I20261119 09:04:14.946350 1271 ts_manager.cc:210] Registered new tserver with Master: c77ec8ad79fa4aeeabda528ce9b285ca (127.30.252.65:44327)
I20261119 09:04:14.947549 1271 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.65:35871
W20261119 09:04:15.047204 1462 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261119 09:04:15.047420 1462 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:15.047453 1462 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:15.047516 1462 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:15.051121 1462 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261119 09:04:15.051182 1462 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261119 09:04:15.051218 1462 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:15.051314 1462 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261119 09:04:15.051360 1462 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.66
I20261119 09:04:15.055446 1462 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.252.66
--webserver_port=0
--tserver_master_addrs=127.30.252.126:41673
--never_fsync=true
--heap_profile_path=/tmp/kudu.1462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:15.056365 1462 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:15.057061 1462 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:15.063105 1468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:15.062992 1467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:15.063210 1470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:15.063330 1462 server_base.cc:1061] running on GCE node
I20261119 09:04:15.063863 1462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:15.064428 1462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:15.065588 1462 hybrid_clock.cc:648] HybridClock initialized: now 1795079055065551 us; error 54 us; skew 500 ppm
I20261119 09:04:15.067281 1462 webserver.cc:577] Webserver started at http://127.30.252.66:41605/ using document root <none> and password file <none>
I20261119 09:04:15.067898 1462 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:15.067951 1462 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:15.068205 1462 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:15.069485 1462 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b49271308e7d416ea0341e0c07f46acc"
format_stamp: "Formatted at 2026-11-19 09:04:15 on dist-test-slave-1fcg"
I20261119 09:04:15.069914 1462 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b49271308e7d416ea0341e0c07f46acc"
format_stamp: "Formatted at 2026-11-19 09:04:15 on dist-test-slave-1fcg"
I20261119 09:04:15.073195 1462 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261119 09:04:15.075686 1476 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:15.076718 1462 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261119 09:04:15.076833 1462 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b49271308e7d416ea0341e0c07f46acc"
format_stamp: "Formatted at 2026-11-19 09:04:15 on dist-test-slave-1fcg"
I20261119 09:04:15.076929 1462 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:15.089504 1462 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:15.089952 1462 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:15.090092 1462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:15.090591 1462 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:15.091642 1462 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:15.091688 1462 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:15.091743 1462 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:15.091775 1462 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:15.100831 1462 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.66:42157
I20261119 09:04:15.100878 1589 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.66:42157 every 8 connection(s)
I20261119 09:04:15.101795 1462 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20261119 09:04:15.109054 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 1462
I20261119 09:04:15.109159 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20261119 09:04:15.110387 1590 heartbeater.cc:345] Connected to a master server at 127.30.252.126:41673
I20261119 09:04:15.110612 1590 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:15.111171 1590 heartbeater.cc:508] Master 127.30.252.126:41673 requested a full tablet report, sending...
I20261119 09:04:15.111413 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.67:0
--local_ip_for_outbound_sockets=127.30.252.67
--webserver_interface=127.30.252.67
--webserver_port=0
--tserver_master_addrs=127.30.252.126:41673
--builtin_ntp_servers=127.30.252.84:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261119 09:04:15.112270 1270 ts_manager.cc:210] Registered new tserver with Master: b49271308e7d416ea0341e0c07f46acc (127.30.252.66:42157)
I20261119 09:04:15.112821 1270 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.66:50085
W20261119 09:04:15.210621 1593 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261119 09:04:15.210825 1593 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:15.210848 1593 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:15.210886 1593 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:15.214509 1593 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261119 09:04:15.214578 1593 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261119 09:04:15.214613 1593 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:15.214677 1593 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261119 09:04:15.214707 1593 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.67
I20261119 09:04:15.218745 1593 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.252.67
--webserver_port=0
--tserver_master_addrs=127.30.252.126:41673
--never_fsync=true
--heap_profile_path=/tmp/kudu.1593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:15.219502 1593 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:15.220238 1593 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:15.226217 1599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:15.226328 1593 server_base.cc:1061] running on GCE node
W20261119 09:04:15.226223 1601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:15.226256 1598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:15.226867 1593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:15.227401 1593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:15.228574 1593 hybrid_clock.cc:648] HybridClock initialized: now 1795079055228538 us; error 49 us; skew 500 ppm
I20261119 09:04:15.230283 1593 webserver.cc:577] Webserver started at http://127.30.252.67:34101/ using document root <none> and password file <none>
I20261119 09:04:15.230847 1593 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:15.230898 1593 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:15.231112 1593 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:15.232461 1593 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "854291848e4c4f7b9675d859a50da59a"
format_stamp: "Formatted at 2026-11-19 09:04:15 on dist-test-slave-1fcg"
I20261119 09:04:15.232869 1593 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "854291848e4c4f7b9675d859a50da59a"
format_stamp: "Formatted at 2026-11-19 09:04:15 on dist-test-slave-1fcg"
I20261119 09:04:15.236259 1593 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20261119 09:04:15.238775 1607 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:15.240159 1593 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20261119 09:04:15.240275 1593 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "854291848e4c4f7b9675d859a50da59a"
format_stamp: "Formatted at 2026-11-19 09:04:15 on dist-test-slave-1fcg"
I20261119 09:04:15.240367 1593 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:15.261480 1593 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:15.262001 1593 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:15.262120 1593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:15.262601 1593 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:15.263607 1593 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:15.263659 1593 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:15.263702 1593 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:15.263715 1593 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:15.272226 1593 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.67:41143
I20261119 09:04:15.272321 1720 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.67:41143 every 8 connection(s)
I20261119 09:04:15.273226 1593 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20261119 09:04:15.277788 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 1593
I20261119 09:04:15.277902 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20261119 09:04:15.280362 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.68:0
--local_ip_for_outbound_sockets=127.30.252.68
--webserver_interface=127.30.252.68
--webserver_port=0
--tserver_master_addrs=127.30.252.126:41673
--builtin_ntp_servers=127.30.252.84:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261119 09:04:15.284018 1721 heartbeater.cc:345] Connected to a master server at 127.30.252.126:41673
I20261119 09:04:15.284289 1721 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:15.284951 1721 heartbeater.cc:508] Master 127.30.252.126:41673 requested a full tablet report, sending...
I20261119 09:04:15.286259 1270 ts_manager.cc:210] Registered new tserver with Master: 854291848e4c4f7b9675d859a50da59a (127.30.252.67:41143)
I20261119 09:04:15.286847 1270 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.67:56141
W20261119 09:04:15.382881 1724 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261119 09:04:15.383107 1724 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:15.383149 1724 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:15.383216 1724 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:15.386788 1724 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261119 09:04:15.386861 1724 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261119 09:04:15.386919 1724 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:15.387017 1724 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261119 09:04:15.387055 1724 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.68
I20261119 09:04:15.391134 1724 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.30.252.68
--webserver_port=0
--tserver_master_addrs=127.30.252.126:41673
--never_fsync=true
--heap_profile_path=/tmp/kudu.1724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.68
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:15.391964 1724 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:15.392652 1724 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:15.398518 1730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:15.398524 1732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:15.399006 1729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:15.399719 1724 server_base.cc:1061] running on GCE node
I20261119 09:04:15.400175 1724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:15.400779 1724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:15.401968 1724 hybrid_clock.cc:648] HybridClock initialized: now 1795079055401946 us; error 36 us; skew 500 ppm
I20261119 09:04:15.403776 1724 webserver.cc:577] Webserver started at http://127.30.252.68:46029/ using document root <none> and password file <none>
I20261119 09:04:15.404304 1724 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:15.404351 1724 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:15.404521 1724 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:15.405776 1724 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "d4fcc7f9fbf54e6897c348825ac49db7"
format_stamp: "Formatted at 2026-11-19 09:04:15 on dist-test-slave-1fcg"
I20261119 09:04:15.406154 1724 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "d4fcc7f9fbf54e6897c348825ac49db7"
format_stamp: "Formatted at 2026-11-19 09:04:15 on dist-test-slave-1fcg"
I20261119 09:04:15.409339 1724 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261119 09:04:15.411653 1738 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:15.412694 1724 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261119 09:04:15.412810 1724 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "d4fcc7f9fbf54e6897c348825ac49db7"
format_stamp: "Formatted at 2026-11-19 09:04:15 on dist-test-slave-1fcg"
I20261119 09:04:15.412894 1724 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:15.424578 1724 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:15.424991 1724 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:15.425110 1724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:15.425583 1724 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:15.426612 1724 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:15.426669 1724 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:15.426712 1724 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:15.426729 1724 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:15.435212 1724 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.68:42929
I20261119 09:04:15.435261 1851 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.68:42929 every 8 connection(s)
I20261119 09:04:15.436219 1724 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20261119 09:04:15.445200 1852 heartbeater.cc:345] Connected to a master server at 127.30.252.126:41673
I20261119 09:04:15.445452 1852 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:15.445954 1852 heartbeater.cc:508] Master 127.30.252.126:41673 requested a full tablet report, sending...
I20261119 09:04:15.446339 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 1724
I20261119 09:04:15.446455 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20261119 09:04:15.447046 1270 ts_manager.cc:210] Registered new tserver with Master: d4fcc7f9fbf54e6897c348825ac49db7 (127.30.252.68:42929)
I20261119 09:04:15.447652 1270 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.68:45445
I20261119 09:04:15.448871 31729 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261119 09:04:15.457036 1270 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40616:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20261119 09:04:15.476137 1393 tablet_service.cc:1495] Processing CreateTablet for tablet 3788b42d5cf245de89903a418c8edbe1 (DEFAULT_TABLE table=TestTable [id=2c1e598e6c68421ea6c83ee88119d57f]), partition=RANGE (key) PARTITION UNBOUNDED
I20261119 09:04:15.476137 1655 tablet_service.cc:1495] Processing CreateTablet for tablet 3788b42d5cf245de89903a418c8edbe1 (DEFAULT_TABLE table=TestTable [id=2c1e598e6c68421ea6c83ee88119d57f]), partition=RANGE (key) PARTITION UNBOUNDED
I20261119 09:04:15.476164 1786 tablet_service.cc:1495] Processing CreateTablet for tablet 3788b42d5cf245de89903a418c8edbe1 (DEFAULT_TABLE table=TestTable [id=2c1e598e6c68421ea6c83ee88119d57f]), partition=RANGE (key) PARTITION UNBOUNDED
I20261119 09:04:15.477099 1393 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3788b42d5cf245de89903a418c8edbe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:15.477123 1655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3788b42d5cf245de89903a418c8edbe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:15.477123 1786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3788b42d5cf245de89903a418c8edbe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:15.482441 1869 tablet_bootstrap.cc:492] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca: Bootstrap starting.
I20261119 09:04:15.483383 1870 tablet_bootstrap.cc:492] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a: Bootstrap starting.
I20261119 09:04:15.483896 1871 tablet_bootstrap.cc:492] T 3788b42d5cf245de89903a418c8edbe1 P d4fcc7f9fbf54e6897c348825ac49db7: Bootstrap starting.
I20261119 09:04:15.485641 1870 tablet_bootstrap.cc:654] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:15.485641 1869 tablet_bootstrap.cc:654] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:15.485900 1871 tablet_bootstrap.cc:654] T 3788b42d5cf245de89903a418c8edbe1 P d4fcc7f9fbf54e6897c348825ac49db7: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:15.486555 1871 log.cc:826] T 3788b42d5cf245de89903a418c8edbe1 P d4fcc7f9fbf54e6897c348825ac49db7: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:15.486543 1869 log.cc:826] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:15.486543 1870 log.cc:826] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:15.487927 1871 tablet_bootstrap.cc:492] T 3788b42d5cf245de89903a418c8edbe1 P d4fcc7f9fbf54e6897c348825ac49db7: No bootstrap required, opened a new log
I20261119 09:04:15.488121 1871 ts_tablet_manager.cc:1449] T 3788b42d5cf245de89903a418c8edbe1 P d4fcc7f9fbf54e6897c348825ac49db7: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20261119 09:04:15.488126 1869 tablet_bootstrap.cc:492] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca: No bootstrap required, opened a new log
I20261119 09:04:15.488126 1870 tablet_bootstrap.cc:492] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a: No bootstrap required, opened a new log
I20261119 09:04:15.488306 1869 ts_tablet_manager.cc:1449] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20261119 09:04:15.488315 1870 ts_tablet_manager.cc:1449] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20261119 09:04:15.490550 1871 raft_consensus.cc:360] T 3788b42d5cf245de89903a418c8edbe1 P d4fcc7f9fbf54e6897c348825ac49db7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854291848e4c4f7b9675d859a50da59a" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 41143 } } peers { permanent_uuid: "d4fcc7f9fbf54e6897c348825ac49db7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42929 } } peers { permanent_uuid: "c77ec8ad79fa4aeeabda528ce9b285ca" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 44327 } }
I20261119 09:04:15.490734 1871 raft_consensus.cc:386] T 3788b42d5cf245de89903a418c8edbe1 P d4fcc7f9fbf54e6897c348825ac49db7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:04:15.490785 1871 raft_consensus.cc:748] T 3788b42d5cf245de89903a418c8edbe1 P d4fcc7f9fbf54e6897c348825ac49db7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4fcc7f9fbf54e6897c348825ac49db7, State: Initialized, Role: FOLLOWER
I20261119 09:04:15.491180 1871 consensus_queue.cc:261] T 3788b42d5cf245de89903a418c8edbe1 P d4fcc7f9fbf54e6897c348825ac49db7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854291848e4c4f7b9675d859a50da59a" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 41143 } } peers { permanent_uuid: "d4fcc7f9fbf54e6897c348825ac49db7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42929 } } peers { permanent_uuid: "c77ec8ad79fa4aeeabda528ce9b285ca" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 44327 } }
I20261119 09:04:15.491392 1870 raft_consensus.cc:360] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854291848e4c4f7b9675d859a50da59a" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 41143 } } peers { permanent_uuid: "d4fcc7f9fbf54e6897c348825ac49db7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42929 } } peers { permanent_uuid: "c77ec8ad79fa4aeeabda528ce9b285ca" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 44327 } }
I20261119 09:04:15.491607 1870 raft_consensus.cc:386] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:04:15.491667 1870 raft_consensus.cc:748] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854291848e4c4f7b9675d859a50da59a, State: Initialized, Role: FOLLOWER
I20261119 09:04:15.491976 1852 heartbeater.cc:500] Master 127.30.252.126:41673 was elected leader, sending a full tablet report...
I20261119 09:04:15.491958 1869 raft_consensus.cc:360] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854291848e4c4f7b9675d859a50da59a" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 41143 } } peers { permanent_uuid: "d4fcc7f9fbf54e6897c348825ac49db7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42929 } } peers { permanent_uuid: "c77ec8ad79fa4aeeabda528ce9b285ca" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 44327 } }
I20261119 09:04:15.492141 1869 raft_consensus.cc:386] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:04:15.492204 1869 raft_consensus.cc:748] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77ec8ad79fa4aeeabda528ce9b285ca, State: Initialized, Role: FOLLOWER
I20261119 09:04:15.492129 1870 consensus_queue.cc:261] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854291848e4c4f7b9675d859a50da59a" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 41143 } } peers { permanent_uuid: "d4fcc7f9fbf54e6897c348825ac49db7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42929 } } peers { permanent_uuid: "c77ec8ad79fa4aeeabda528ce9b285ca" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 44327 } }
I20261119 09:04:15.492347 1871 ts_tablet_manager.cc:1480] T 3788b42d5cf245de89903a418c8edbe1 P d4fcc7f9fbf54e6897c348825ac49db7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20261119 09:04:15.492638 1869 consensus_queue.cc:261] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854291848e4c4f7b9675d859a50da59a" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 41143 } } peers { permanent_uuid: "d4fcc7f9fbf54e6897c348825ac49db7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42929 } } peers { permanent_uuid: "c77ec8ad79fa4aeeabda528ce9b285ca" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 44327 } }
I20261119 09:04:15.492900 1870 ts_tablet_manager.cc:1480] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20261119 09:04:15.493250 1459 heartbeater.cc:500] Master 127.30.252.126:41673 was elected leader, sending a full tablet report...
I20261119 09:04:15.493321 1869 ts_tablet_manager.cc:1480] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20261119 09:04:15.493861 1721 heartbeater.cc:500] Master 127.30.252.126:41673 was elected leader, sending a full tablet report...
W20261119 09:04:15.537138 1853 tablet.cc:2554] T 3788b42d5cf245de89903a418c8edbe1 P d4fcc7f9fbf54e6897c348825ac49db7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20261119 09:04:15.537148 1460 tablet.cc:2554] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20261119 09:04:15.574456 1722 tablet.cc:2554] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261119 09:04:15.719337 1877 raft_consensus.cc:499] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261119 09:04:15.719553 1877 raft_consensus.cc:521] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854291848e4c4f7b9675d859a50da59a" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 41143 } } peers { permanent_uuid: "d4fcc7f9fbf54e6897c348825ac49db7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42929 } } peers { permanent_uuid: "c77ec8ad79fa4aeeabda528ce9b285ca" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 44327 } }
I20261119 09:04:15.720840 1877 leader_election.cc:290] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 854291848e4c4f7b9675d859a50da59a (127.30.252.67:41143), d4fcc7f9fbf54e6897c348825ac49db7 (127.30.252.68:42929)
I20261119 09:04:15.727210 1675 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "3788b42d5cf245de89903a418c8edbe1" candidate_uuid: "c77ec8ad79fa4aeeabda528ce9b285ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854291848e4c4f7b9675d859a50da59a" is_pre_election: true
I20261119 09:04:15.727461 1806 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "3788b42d5cf245de89903a418c8edbe1" candidate_uuid: "c77ec8ad79fa4aeeabda528ce9b285ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4fcc7f9fbf54e6897c348825ac49db7" is_pre_election: true
I20261119 09:04:15.727617 1675 raft_consensus.cc:2490] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c77ec8ad79fa4aeeabda528ce9b285ca in term 0.
I20261119 09:04:15.727710 1806 raft_consensus.cc:2490] T 3788b42d5cf245de89903a418c8edbe1 P d4fcc7f9fbf54e6897c348825ac49db7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c77ec8ad79fa4aeeabda528ce9b285ca in term 0.
I20261119 09:04:15.728143 1346 leader_election.cc:304] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 854291848e4c4f7b9675d859a50da59a, c77ec8ad79fa4aeeabda528ce9b285ca; no voters:
I20261119 09:04:15.728432 1877 raft_consensus.cc:2826] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261119 09:04:15.728502 1877 raft_consensus.cc:499] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261119 09:04:15.728534 1877 raft_consensus.cc:3096] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:15.729467 1877 raft_consensus.cc:521] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854291848e4c4f7b9675d859a50da59a" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 41143 } } peers { permanent_uuid: "d4fcc7f9fbf54e6897c348825ac49db7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42929 } } peers { permanent_uuid: "c77ec8ad79fa4aeeabda528ce9b285ca" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 44327 } }
I20261119 09:04:15.730166 1877 leader_election.cc:290] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca [CANDIDATE]: Term 1 election: Requested vote from peers 854291848e4c4f7b9675d859a50da59a (127.30.252.67:41143), d4fcc7f9fbf54e6897c348825ac49db7 (127.30.252.68:42929)
I20261119 09:04:15.730628 1675 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "3788b42d5cf245de89903a418c8edbe1" candidate_uuid: "c77ec8ad79fa4aeeabda528ce9b285ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854291848e4c4f7b9675d859a50da59a"
I20261119 09:04:15.730752 1675 raft_consensus.cc:3096] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:15.730724 1806 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "3788b42d5cf245de89903a418c8edbe1" candidate_uuid: "c77ec8ad79fa4aeeabda528ce9b285ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4fcc7f9fbf54e6897c348825ac49db7"
I20261119 09:04:15.730893 1806 raft_consensus.cc:3096] T 3788b42d5cf245de89903a418c8edbe1 P d4fcc7f9fbf54e6897c348825ac49db7 [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:15.731716 1675 raft_consensus.cc:2490] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c77ec8ad79fa4aeeabda528ce9b285ca in term 1.
I20261119 09:04:15.731786 1806 raft_consensus.cc:2490] T 3788b42d5cf245de89903a418c8edbe1 P d4fcc7f9fbf54e6897c348825ac49db7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c77ec8ad79fa4aeeabda528ce9b285ca in term 1.
I20261119 09:04:15.732010 1346 leader_election.cc:304] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 854291848e4c4f7b9675d859a50da59a, c77ec8ad79fa4aeeabda528ce9b285ca; no voters:
I20261119 09:04:15.732246 1877 raft_consensus.cc:2826] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca [term 1 FOLLOWER]: Leader election won for term 1
I20261119 09:04:15.732475 1877 raft_consensus.cc:705] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca [term 1 LEADER]: Becoming Leader. State: Replica: c77ec8ad79fa4aeeabda528ce9b285ca, State: Running, Role: LEADER
I20261119 09:04:15.732833 1877 consensus_queue.cc:238] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854291848e4c4f7b9675d859a50da59a" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 41143 } } peers { permanent_uuid: "d4fcc7f9fbf54e6897c348825ac49db7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42929 } } peers { permanent_uuid: "c77ec8ad79fa4aeeabda528ce9b285ca" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 44327 } }
I20261119 09:04:15.736274 1880 consensus_queue.cc:1049] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "854291848e4c4f7b9675d859a50da59a" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 41143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261119 09:04:15.736613 1877 consensus_queue.cc:1049] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4fcc7f9fbf54e6897c348825ac49db7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261119 09:04:15.736578 1269 catalog_manager.cc:5856] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca reported cstate change: term changed from 0 to 1, leader changed from <none> to c77ec8ad79fa4aeeabda528ce9b285ca (127.30.252.65). New cstate: current_term: 1 leader_uuid: "c77ec8ad79fa4aeeabda528ce9b285ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854291848e4c4f7b9675d859a50da59a" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 41143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4fcc7f9fbf54e6897c348825ac49db7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c77ec8ad79fa4aeeabda528ce9b285ca" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 44327 } health_report { overall_health: HEALTHY } } }
I20261119 09:04:15.763597 31729 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261119 09:04:15.765645 31729 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c77ec8ad79fa4aeeabda528ce9b285ca to finish bootstrapping
I20261119 09:04:15.771221 31729 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 854291848e4c4f7b9675d859a50da59a to finish bootstrapping
I20261119 09:04:15.776424 31729 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d4fcc7f9fbf54e6897c348825ac49db7 to finish bootstrapping
I20261119 09:04:15.781493 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 1593
I20261119 09:04:15.787784 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.67:41143
--local_ip_for_outbound_sockets=127.30.252.67
--tserver_master_addrs=127.30.252.126:41673
--webserver_port=34101
--webserver_interface=127.30.252.67
--builtin_ntp_servers=127.30.252.84:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000
--tablet_bootstrap_inject_latency_ms=4000 with env {}
W20261119 09:04:15.893033 1889 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261119 09:04:15.893237 1889 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:15.893265 1889 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:15.893308 1889 flags.cc:432] Enabled unsafe flag: --tablet_bootstrap_inject_latency_ms=4000
W20261119 09:04:15.893330 1889 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:15.896808 1889 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261119 09:04:15.896865 1889 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261119 09:04:15.896900 1889 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:15.896970 1889 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261119 09:04:15.897001 1889 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.67
I20261119 09:04:15.901181 1889 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.67:41143
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.252.67
--webserver_port=34101
--tserver_master_addrs=127.30.252.126:41673
--tablet_bootstrap_inject_latency_ms=4000
--never_fsync=true
--heap_profile_path=/tmp/kudu.1889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:15.902020 1889 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:15.902727 1889 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:15.909094 1895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:15.909114 1894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:15.909103 1897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:15.909441 1889 server_base.cc:1061] running on GCE node
I20261119 09:04:15.909788 1889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:15.910434 1889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:15.911638 1889 hybrid_clock.cc:648] HybridClock initialized: now 1795079055911612 us; error 38 us; skew 500 ppm
I20261119 09:04:15.913344 1889 webserver.cc:577] Webserver started at http://127.30.252.67:34101/ using document root <none> and password file <none>
I20261119 09:04:15.913911 1889 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:15.913982 1889 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:15.917876 1889 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20261119 09:04:15.920295 1903 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:15.921401 1889 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261119 09:04:15.921525 1889 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "854291848e4c4f7b9675d859a50da59a"
format_stamp: "Formatted at 2026-11-19 09:04:15 on dist-test-slave-1fcg"
I20261119 09:04:15.922014 1889 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:15.966473 1889 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:15.966933 1889 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:15.967073 1889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:15.967620 1889 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:15.968822 1910 ts_tablet_manager.cc:588] Loading tablet metadata (0/1 complete)
I20261119 09:04:15.972321 1889 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261119 09:04:15.972383 1889 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20261119 09:04:15.972448 1889 ts_tablet_manager.cc:646] Registering tablets (0/1 complete)
I20261119 09:04:15.973932 1889 ts_tablet_manager.cc:662] Registered 1 tablets
W20261119 09:04:15.973987 1910 ts_tablet_manager.cc:1428] Injecting 4000ms delay in tablet bootstrapping
I20261119 09:04:15.973986 1889 ts_tablet_manager.cc:641] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20261119 09:04:15.983701 1889 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.67:41143
I20261119 09:04:15.983744 2017 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.67:41143 every 8 connection(s)
I20261119 09:04:15.984762 1889 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20261119 09:04:15.992337 2018 heartbeater.cc:345] Connected to a master server at 127.30.252.126:41673
I20261119 09:04:15.992589 2018 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:15.993060 2018 heartbeater.cc:508] Master 127.30.252.126:41673 requested a full tablet report, sending...
I20261119 09:04:15.994601 1271 ts_manager.cc:210] Re-registered known tserver with Master: 854291848e4c4f7b9675d859a50da59a (127.30.252.67:41143)
I20261119 09:04:15.994692 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 1889
I20261119 09:04:15.995357 1271 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.67:59671
I20261119 09:04:16.115029 1590 heartbeater.cc:500] Master 127.30.252.126:41673 was elected leader, sending a full tablet report...
W20261119 09:04:16.297375 1346 consensus_peers.cc:597] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca -> Peer 854291848e4c4f7b9675d859a50da59a (127.30.252.67:41143): Couldn't send request to peer 854291848e4c4f7b9675d859a50da59a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20261119 09:04:16.997494 2018 heartbeater.cc:500] Master 127.30.252.126:41673 was elected leader, sending a full tablet report...
W20261119 09:04:18.829113 1346 consensus_peers.cc:597] T 3788b42d5cf245de89903a418c8edbe1 P c77ec8ad79fa4aeeabda528ce9b285ca -> Peer 854291848e4c4f7b9675d859a50da59a (127.30.252.67:41143): Couldn't send request to peer 854291848e4c4f7b9675d859a50da59a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20261119 09:04:19.974766 1910 tablet_bootstrap.cc:492] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a: Bootstrap starting.
I20261119 09:04:19.984091 1910 log.cc:826] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:19.987011 1910 tablet_bootstrap.cc:492] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261119 09:04:19.987375 1910 tablet_bootstrap.cc:492] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a: Bootstrap complete.
I20261119 09:04:19.987877 1910 ts_tablet_manager.cc:1449] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.004s
I20261119 09:04:19.990589 1910 raft_consensus.cc:360] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854291848e4c4f7b9675d859a50da59a" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 41143 } } peers { permanent_uuid: "d4fcc7f9fbf54e6897c348825ac49db7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42929 } } peers { permanent_uuid: "c77ec8ad79fa4aeeabda528ce9b285ca" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 44327 } }
I20261119 09:04:19.990995 1910 raft_consensus.cc:748] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854291848e4c4f7b9675d859a50da59a, State: Initialized, Role: FOLLOWER
I20261119 09:04:19.991499 1910 consensus_queue.cc:261] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854291848e4c4f7b9675d859a50da59a" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 41143 } } peers { permanent_uuid: "d4fcc7f9fbf54e6897c348825ac49db7" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 42929 } } peers { permanent_uuid: "c77ec8ad79fa4aeeabda528ce9b285ca" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 44327 } }
I20261119 09:04:19.992406 1910 ts_tablet_manager.cc:1480] T 3788b42d5cf245de89903a418c8edbe1 P 854291848e4c4f7b9675d859a50da59a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20261119 09:04:23.023265 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 1331
I20261119 09:04:23.030215 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 1462
I20261119 09:04:23.035637 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 1889
I20261119 09:04:23.041083 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 1724
I20261119 09:04:23.046609 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 1240
2026-11-19T09:04:23Z chronyd exiting
[ OK ] SlowTabletBootstrapTest.TestSlowBootstrap (8571 ms)
[ RUN ] SlowTabletBootstrapTest.TestFallBehind
I20261119 09:04:23.069258 31729 ts_itest-base.cc:115] Starting cluster with:
I20261119 09:04:23.069305 31729 ts_itest-base.cc:116] --------------
I20261119 09:04:23.069319 31729 ts_itest-base.cc:117] 4 tablet servers
I20261119 09:04:23.069332 31729 ts_itest-base.cc:118] 3 replicas per TS
I20261119 09:04:23.069345 31729 ts_itest-base.cc:119] --------------
2026-11-19T09:04:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-11-19T09:04:23Z Disabled control of system clock
I20261119 09:04:23.078379 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.252.126:40941
--webserver_interface=127.30.252.126
--webserver_port=0
--builtin_ntp_servers=127.30.252.84:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.252.126:40941 with env {}
W20261119 09:04:23.189425 2039 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:23.189625 2039 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:23.189683 2039 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:23.193511 2039 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261119 09:04:23.193590 2039 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:23.193624 2039 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261119 09:04:23.193650 2039 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261119 09:04:23.198778 2039 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.252.126:40941
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.252.126:40941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.252.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:23.199963 2039 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:23.200731 2039 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:23.207876 2047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:23.208170 2039 server_base.cc:1061] running on GCE node
W20261119 09:04:23.208786 2044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:23.208796 2045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:23.209467 2039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:23.210398 2039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:23.211616 2039 hybrid_clock.cc:648] HybridClock initialized: now 1795079063211567 us; error 56 us; skew 500 ppm
I20261119 09:04:23.213431 2039 webserver.cc:577] Webserver started at http://127.30.252.126:38171/ using document root <none> and password file <none>
I20261119 09:04:23.214057 2039 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:23.214107 2039 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:23.214316 2039 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:23.215879 2039 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ff15b8d5a9cd40abbdf53c4f246f77e5"
format_stamp: "Formatted at 2026-11-19 09:04:23 on dist-test-slave-1fcg"
I20261119 09:04:23.216351 2039 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ff15b8d5a9cd40abbdf53c4f246f77e5"
format_stamp: "Formatted at 2026-11-19 09:04:23 on dist-test-slave-1fcg"
I20261119 09:04:23.220583 2039 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20261119 09:04:23.223639 2053 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:23.224877 2039 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20261119 09:04:23.225018 2039 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ff15b8d5a9cd40abbdf53c4f246f77e5"
format_stamp: "Formatted at 2026-11-19 09:04:23 on dist-test-slave-1fcg"
I20261119 09:04:23.225142 2039 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:23.243448 2039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:23.244047 2039 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:23.244185 2039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:23.251467 2039 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.126:40941
I20261119 09:04:23.251473 2105 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.126:40941 every 8 connection(s)
I20261119 09:04:23.252588 2039 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20261119 09:04:23.254765 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 2039
I20261119 09:04:23.254894 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal/instance
I20261119 09:04:23.256379 2106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:23.262514 2106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5: Bootstrap starting.
I20261119 09:04:23.265177 2106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:23.266026 2106 log.cc:826] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:23.267820 2106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5: No bootstrap required, opened a new log
I20261119 09:04:23.270571 2106 raft_consensus.cc:360] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff15b8d5a9cd40abbdf53c4f246f77e5" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 40941 } }
I20261119 09:04:23.270776 2106 raft_consensus.cc:386] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:04:23.270819 2106 raft_consensus.cc:748] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff15b8d5a9cd40abbdf53c4f246f77e5, State: Initialized, Role: FOLLOWER
I20261119 09:04:23.271216 2106 consensus_queue.cc:261] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff15b8d5a9cd40abbdf53c4f246f77e5" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 40941 } }
I20261119 09:04:23.271359 2106 raft_consensus.cc:400] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261119 09:04:23.271404 2106 raft_consensus.cc:499] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261119 09:04:23.271497 2106 raft_consensus.cc:3096] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5 [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:23.272897 2106 raft_consensus.cc:521] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff15b8d5a9cd40abbdf53c4f246f77e5" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 40941 } }
I20261119 09:04:23.273257 2106 leader_election.cc:304] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff15b8d5a9cd40abbdf53c4f246f77e5; no voters:
I20261119 09:04:23.273542 2106 leader_election.cc:290] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261119 09:04:23.273676 2111 raft_consensus.cc:2826] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5 [term 1 FOLLOWER]: Leader election won for term 1
I20261119 09:04:23.273873 2111 raft_consensus.cc:705] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5 [term 1 LEADER]: Becoming Leader. State: Replica: ff15b8d5a9cd40abbdf53c4f246f77e5, State: Running, Role: LEADER
I20261119 09:04:23.274260 2111 consensus_queue.cc:238] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff15b8d5a9cd40abbdf53c4f246f77e5" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 40941 } }
I20261119 09:04:23.274674 2106 sys_catalog.cc:565] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5 [sys.catalog]: configured and running, proceeding with master startup.
I20261119 09:04:23.275854 2112 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff15b8d5a9cd40abbdf53c4f246f77e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff15b8d5a9cd40abbdf53c4f246f77e5" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 40941 } } }
I20261119 09:04:23.275977 2112 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5 [sys.catalog]: This master's current role is: LEADER
I20261119 09:04:23.276371 2120 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261119 09:04:23.277648 2113 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff15b8d5a9cd40abbdf53c4f246f77e5. Latest consensus state: current_term: 1 leader_uuid: "ff15b8d5a9cd40abbdf53c4f246f77e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff15b8d5a9cd40abbdf53c4f246f77e5" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 40941 } } }
I20261119 09:04:23.277733 2113 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5 [sys.catalog]: This master's current role is: LEADER
I20261119 09:04:23.280143 2120 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261119 09:04:23.286244 2120 catalog_manager.cc:1398] Generated new cluster ID: 967a3b156bcd4b57af4f5a532900fb82
I20261119 09:04:23.286417 2120 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261119 09:04:23.300207 2120 catalog_manager.cc:1421] Generated new certificate authority record
I20261119 09:04:23.301414 2120 catalog_manager.cc:1555] Loading token signing keys...
I20261119 09:04:23.311003 2120 catalog_manager.cc:6237] T 00000000000000000000000000000000 P ff15b8d5a9cd40abbdf53c4f246f77e5: Generated new TSK 0
I20261119 09:04:23.311873 2120 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261119 09:04:23.323002 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.65:0
--local_ip_for_outbound_sockets=127.30.252.65
--webserver_interface=127.30.252.65
--webserver_port=0
--tserver_master_addrs=127.30.252.126:40941
--builtin_ntp_servers=127.30.252.84:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20261119 09:04:23.427807 2130 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261119 09:04:23.428025 2130 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:23.428064 2130 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:23.428130 2130 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:23.431912 2130 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261119 09:04:23.431984 2130 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261119 09:04:23.432240 2130 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:23.432353 2130 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261119 09:04:23.432394 2130 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.65
I20261119 09:04:23.437179 2130 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.252.65
--webserver_port=0
--tserver_master_addrs=127.30.252.126:40941
--never_fsync=true
--heap_profile_path=/tmp/kudu.2130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:23.438117 2130 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:23.438951 2130 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:23.445847 2135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:23.446017 2138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:23.446265 2136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:23.446324 2130 server_base.cc:1061] running on GCE node
I20261119 09:04:23.446730 2130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:23.447296 2130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:23.448485 2130 hybrid_clock.cc:648] HybridClock initialized: now 1795079063448455 us; error 42 us; skew 500 ppm
I20261119 09:04:23.450451 2130 webserver.cc:577] Webserver started at http://127.30.252.65:36025/ using document root <none> and password file <none>
I20261119 09:04:23.451027 2130 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:23.451076 2130 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:23.451288 2130 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:23.452642 2130 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3e06e721e65e492891128523e5df549d"
format_stamp: "Formatted at 2026-11-19 09:04:23 on dist-test-slave-1fcg"
I20261119 09:04:23.453111 2130 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3e06e721e65e492891128523e5df549d"
format_stamp: "Formatted at 2026-11-19 09:04:23 on dist-test-slave-1fcg"
I20261119 09:04:23.456897 2130 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20261119 09:04:23.459873 2144 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:23.461187 2130 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20261119 09:04:23.461311 2130 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3e06e721e65e492891128523e5df549d"
format_stamp: "Formatted at 2026-11-19 09:04:23 on dist-test-slave-1fcg"
I20261119 09:04:23.461410 2130 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:23.470491 2130 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:23.470964 2130 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:23.471103 2130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:23.471624 2130 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:23.472620 2130 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:23.472667 2130 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:23.472721 2130 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:23.472757 2130 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:23.482916 2130 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.65:46315
I20261119 09:04:23.482946 2258 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.65:46315 every 8 connection(s)
I20261119 09:04:23.483995 2130 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20261119 09:04:23.488823 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 2130
I20261119 09:04:23.488960 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20261119 09:04:23.491521 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.66:0
--local_ip_for_outbound_sockets=127.30.252.66
--webserver_interface=127.30.252.66
--webserver_port=0
--tserver_master_addrs=127.30.252.126:40941
--builtin_ntp_servers=127.30.252.84:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261119 09:04:23.492640 2259 heartbeater.cc:345] Connected to a master server at 127.30.252.126:40941
I20261119 09:04:23.492849 2259 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:23.493353 2259 heartbeater.cc:508] Master 127.30.252.126:40941 requested a full tablet report, sending...
I20261119 09:04:23.494937 2069 ts_manager.cc:210] Registered new tserver with Master: 3e06e721e65e492891128523e5df549d (127.30.252.65:46315)
I20261119 09:04:23.496476 2069 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.65:60279
W20261119 09:04:23.604436 2262 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261119 09:04:23.604718 2262 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:23.604761 2262 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:23.604827 2262 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:23.608719 2262 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261119 09:04:23.608796 2262 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261119 09:04:23.608846 2262 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:23.608939 2262 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261119 09:04:23.608978 2262 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.66
I20261119 09:04:23.613307 2262 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.252.66
--webserver_port=0
--tserver_master_addrs=127.30.252.126:40941
--never_fsync=true
--heap_profile_path=/tmp/kudu.2262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:23.614166 2262 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:23.615027 2262 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:23.622481 2267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:23.622488 2268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:23.622479 2270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:23.622818 2262 server_base.cc:1061] running on GCE node
I20261119 09:04:23.623786 2262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:23.624797 2262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:23.626055 2262 hybrid_clock.cc:648] HybridClock initialized: now 1795079063626028 us; error 43 us; skew 500 ppm
I20261119 09:04:23.628228 2262 webserver.cc:577] Webserver started at http://127.30.252.66:36029/ using document root <none> and password file <none>
I20261119 09:04:23.628875 2262 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:23.629010 2262 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:23.629302 2262 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:23.630649 2262 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "00eb8e39eccd49f1ad6d989afb2fdd88"
format_stamp: "Formatted at 2026-11-19 09:04:23 on dist-test-slave-1fcg"
I20261119 09:04:23.631049 2262 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "00eb8e39eccd49f1ad6d989afb2fdd88"
format_stamp: "Formatted at 2026-11-19 09:04:23 on dist-test-slave-1fcg"
I20261119 09:04:23.634502 2262 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261119 09:04:23.637573 2276 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:23.639040 2262 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20261119 09:04:23.639163 2262 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "00eb8e39eccd49f1ad6d989afb2fdd88"
format_stamp: "Formatted at 2026-11-19 09:04:23 on dist-test-slave-1fcg"
I20261119 09:04:23.639278 2262 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:23.668618 2262 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:23.669077 2262 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:23.669201 2262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:23.669726 2262 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:23.670778 2262 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:23.670825 2262 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:23.670877 2262 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:23.670895 2262 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:23.681751 2262 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.66:34869
I20261119 09:04:23.681840 2389 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.66:34869 every 8 connection(s)
I20261119 09:04:23.682765 2262 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20261119 09:04:23.689440 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 2262
I20261119 09:04:23.689553 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20261119 09:04:23.692155 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.67:0
--local_ip_for_outbound_sockets=127.30.252.67
--webserver_interface=127.30.252.67
--webserver_port=0
--tserver_master_addrs=127.30.252.126:40941
--builtin_ntp_servers=127.30.252.84:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261119 09:04:23.695194 2390 heartbeater.cc:345] Connected to a master server at 127.30.252.126:40941
I20261119 09:04:23.695466 2390 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:23.696169 2390 heartbeater.cc:508] Master 127.30.252.126:40941 requested a full tablet report, sending...
I20261119 09:04:23.697665 2069 ts_manager.cc:210] Registered new tserver with Master: 00eb8e39eccd49f1ad6d989afb2fdd88 (127.30.252.66:34869)
I20261119 09:04:23.698539 2069 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.66:47837
W20261119 09:04:23.801893 2393 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261119 09:04:23.802109 2393 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:23.802134 2393 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:23.802201 2393 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:23.805744 2393 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261119 09:04:23.805810 2393 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261119 09:04:23.805848 2393 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:23.805949 2393 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261119 09:04:23.805995 2393 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.67
I20261119 09:04:23.810102 2393 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.252.67
--webserver_port=0
--tserver_master_addrs=127.30.252.126:40941
--never_fsync=true
--heap_profile_path=/tmp/kudu.2393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:23.811050 2393 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:23.811820 2393 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:23.817462 2399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:23.817524 2393 server_base.cc:1061] running on GCE node
W20261119 09:04:23.817454 2401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:23.817430 2398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:23.817974 2393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:23.818507 2393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:23.819658 2393 hybrid_clock.cc:648] HybridClock initialized: now 1795079063819638 us; error 33 us; skew 500 ppm
I20261119 09:04:23.821308 2393 webserver.cc:577] Webserver started at http://127.30.252.67:40245/ using document root <none> and password file <none>
I20261119 09:04:23.821843 2393 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:23.821897 2393 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:23.822093 2393 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:23.823369 2393 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ae60e0d407284144aa0861df39608d35"
format_stamp: "Formatted at 2026-11-19 09:04:23 on dist-test-slave-1fcg"
I20261119 09:04:23.823778 2393 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ae60e0d407284144aa0861df39608d35"
format_stamp: "Formatted at 2026-11-19 09:04:23 on dist-test-slave-1fcg"
I20261119 09:04:23.827025 2393 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20261119 09:04:23.830011 2407 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:23.831265 2393 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261119 09:04:23.831394 2393 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ae60e0d407284144aa0861df39608d35"
format_stamp: "Formatted at 2026-11-19 09:04:23 on dist-test-slave-1fcg"
I20261119 09:04:23.831487 2393 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:23.854420 2393 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:23.854873 2393 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:23.855010 2393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:23.855494 2393 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:23.856458 2393 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:23.856509 2393 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:23.856563 2393 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:23.856591 2393 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:23.864989 2393 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.67:44273
I20261119 09:04:23.865031 2520 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.67:44273 every 8 connection(s)
I20261119 09:04:23.866005 2393 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20261119 09:04:23.868913 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 2393
I20261119 09:04:23.869015 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20261119 09:04:23.871121 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.68:0
--local_ip_for_outbound_sockets=127.30.252.68
--webserver_interface=127.30.252.68
--webserver_port=0
--tserver_master_addrs=127.30.252.126:40941
--builtin_ntp_servers=127.30.252.84:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261119 09:04:23.873579 2521 heartbeater.cc:345] Connected to a master server at 127.30.252.126:40941
I20261119 09:04:23.873811 2521 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:23.874393 2521 heartbeater.cc:508] Master 127.30.252.126:40941 requested a full tablet report, sending...
I20261119 09:04:23.875460 2069 ts_manager.cc:210] Registered new tserver with Master: ae60e0d407284144aa0861df39608d35 (127.30.252.67:44273)
I20261119 09:04:23.876049 2069 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.67:35239
W20261119 09:04:23.975243 2524 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261119 09:04:23.975497 2524 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:23.975540 2524 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:23.975616 2524 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:23.979491 2524 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261119 09:04:23.979612 2524 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261119 09:04:23.979676 2524 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:23.979780 2524 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261119 09:04:23.979820 2524 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.68
I20261119 09:04:23.983953 2524 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.30.252.68
--webserver_port=0
--tserver_master_addrs=127.30.252.126:40941
--never_fsync=true
--heap_profile_path=/tmp/kudu.2524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.68
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:23.984789 2524 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:23.985575 2524 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:23.991796 2529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:23.991837 2530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:23.991878 2532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:23.991818 2524 server_base.cc:1061] running on GCE node
I20261119 09:04:23.992544 2524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:23.993094 2524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:23.994325 2524 hybrid_clock.cc:648] HybridClock initialized: now 1795079063994284 us; error 55 us; skew 500 ppm
I20261119 09:04:23.996208 2524 webserver.cc:577] Webserver started at http://127.30.252.68:42991/ using document root <none> and password file <none>
I20261119 09:04:23.996953 2524 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:23.997016 2524 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:23.997249 2524 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:23.998543 2524 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "a9ef1f150cf7437eba4a78244f9cafb5"
format_stamp: "Formatted at 2026-11-19 09:04:23 on dist-test-slave-1fcg"
I20261119 09:04:23.999101 2524 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "a9ef1f150cf7437eba4a78244f9cafb5"
format_stamp: "Formatted at 2026-11-19 09:04:23 on dist-test-slave-1fcg"
I20261119 09:04:24.002547 2524 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20261119 09:04:24.004994 2538 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:24.006076 2524 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261119 09:04:24.006186 2524 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "a9ef1f150cf7437eba4a78244f9cafb5"
format_stamp: "Formatted at 2026-11-19 09:04:23 on dist-test-slave-1fcg"
I20261119 09:04:24.006264 2524 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:24.016466 2524 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:24.016907 2524 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:24.017047 2524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:24.017527 2524 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:24.018419 2524 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:24.018474 2524 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:24.018514 2524 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:24.018527 2524 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:24.028120 2524 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.68:45669
I20261119 09:04:24.028160 2651 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.68:45669 every 8 connection(s)
I20261119 09:04:24.029121 2524 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20261119 09:04:24.036978 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 2524
I20261119 09:04:24.037094 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20261119 09:04:24.037706 2652 heartbeater.cc:345] Connected to a master server at 127.30.252.126:40941
I20261119 09:04:24.037955 2652 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:24.038544 2652 heartbeater.cc:508] Master 127.30.252.126:40941 requested a full tablet report, sending...
I20261119 09:04:24.040141 2069 ts_manager.cc:210] Registered new tserver with Master: a9ef1f150cf7437eba4a78244f9cafb5 (127.30.252.68:45669)
I20261119 09:04:24.040898 2069 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.68:55095
I20261119 09:04:24.052541 31729 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261119 09:04:24.062300 2069 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37430:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20261119 09:04:24.081588 2324 tablet_service.cc:1495] Processing CreateTablet for tablet 82ce84dd8a11441d88584a275c11e8c2 (DEFAULT_TABLE table=TestTable [id=fc6e5efd035c4f7f93c00c1d288944a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20261119 09:04:24.081661 2455 tablet_service.cc:1495] Processing CreateTablet for tablet 82ce84dd8a11441d88584a275c11e8c2 (DEFAULT_TABLE table=TestTable [id=fc6e5efd035c4f7f93c00c1d288944a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20261119 09:04:24.081661 2585 tablet_service.cc:1495] Processing CreateTablet for tablet 82ce84dd8a11441d88584a275c11e8c2 (DEFAULT_TABLE table=TestTable [id=fc6e5efd035c4f7f93c00c1d288944a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20261119 09:04:24.082643 2455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82ce84dd8a11441d88584a275c11e8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:24.082643 2585 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82ce84dd8a11441d88584a275c11e8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:24.082710 2324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82ce84dd8a11441d88584a275c11e8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:24.087967 2669 tablet_bootstrap.cc:492] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88: Bootstrap starting.
I20261119 09:04:24.088928 2670 tablet_bootstrap.cc:492] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5: Bootstrap starting.
I20261119 09:04:24.089547 2671 tablet_bootstrap.cc:492] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35: Bootstrap starting.
I20261119 09:04:24.091147 2670 tablet_bootstrap.cc:654] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:24.091148 2669 tablet_bootstrap.cc:654] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:24.091364 2671 tablet_bootstrap.cc:654] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:24.092096 2671 log.cc:826] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:24.092113 2670 log.cc:826] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:24.092113 2669 log.cc:826] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:24.093528 2670 tablet_bootstrap.cc:492] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5: No bootstrap required, opened a new log
I20261119 09:04:24.093643 2669 tablet_bootstrap.cc:492] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88: No bootstrap required, opened a new log
I20261119 09:04:24.093643 2671 tablet_bootstrap.cc:492] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35: No bootstrap required, opened a new log
I20261119 09:04:24.093760 2670 ts_tablet_manager.cc:1449] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20261119 09:04:24.093850 2669 ts_tablet_manager.cc:1449] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20261119 09:04:24.093850 2671 ts_tablet_manager.cc:1449] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20261119 09:04:24.096508 2670 raft_consensus.cc:360] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00eb8e39eccd49f1ad6d989afb2fdd88" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 34869 } } peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } }
I20261119 09:04:24.096704 2670 raft_consensus.cc:386] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:04:24.096768 2670 raft_consensus.cc:748] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ef1f150cf7437eba4a78244f9cafb5, State: Initialized, Role: FOLLOWER
I20261119 09:04:24.097129 2669 raft_consensus.cc:360] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00eb8e39eccd49f1ad6d989afb2fdd88" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 34869 } } peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } }
I20261119 09:04:24.097129 2671 raft_consensus.cc:360] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00eb8e39eccd49f1ad6d989afb2fdd88" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 34869 } } peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } }
I20261119 09:04:24.097342 2669 raft_consensus.cc:386] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:04:24.097369 2671 raft_consensus.cc:386] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:04:24.097406 2669 raft_consensus.cc:748] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00eb8e39eccd49f1ad6d989afb2fdd88, State: Initialized, Role: FOLLOWER
I20261119 09:04:24.097414 2671 raft_consensus.cc:748] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae60e0d407284144aa0861df39608d35, State: Initialized, Role: FOLLOWER
I20261119 09:04:24.097426 2670 consensus_queue.cc:261] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00eb8e39eccd49f1ad6d989afb2fdd88" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 34869 } } peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } }
I20261119 09:04:24.097805 2671 consensus_queue.cc:261] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00eb8e39eccd49f1ad6d989afb2fdd88" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 34869 } } peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } }
I20261119 09:04:24.097805 2669 consensus_queue.cc:261] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00eb8e39eccd49f1ad6d989afb2fdd88" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 34869 } } peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } }
I20261119 09:04:24.098207 2652 heartbeater.cc:500] Master 127.30.252.126:40941 was elected leader, sending a full tablet report...
I20261119 09:04:24.098304 2670 ts_tablet_manager.cc:1480] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20261119 09:04:24.098548 2669 ts_tablet_manager.cc:1480] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20261119 09:04:24.098548 2671 ts_tablet_manager.cc:1480] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20261119 09:04:24.099191 2521 heartbeater.cc:500] Master 127.30.252.126:40941 was elected leader, sending a full tablet report...
I20261119 09:04:24.099272 2390 heartbeater.cc:500] Master 127.30.252.126:40941 was elected leader, sending a full tablet report...
W20261119 09:04:24.129957 2653 tablet.cc:2554] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261119 09:04:24.158282 2677 raft_consensus.cc:499] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261119 09:04:24.158483 2677 raft_consensus.cc:521] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00eb8e39eccd49f1ad6d989afb2fdd88" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 34869 } } peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } }
I20261119 09:04:24.159933 2677 leader_election.cc:290] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00eb8e39eccd49f1ad6d989afb2fdd88 (127.30.252.66:34869), a9ef1f150cf7437eba4a78244f9cafb5 (127.30.252.68:45669)
I20261119 09:04:24.165364 2604 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "82ce84dd8a11441d88584a275c11e8c2" candidate_uuid: "ae60e0d407284144aa0861df39608d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" is_pre_election: true
I20261119 09:04:24.165359 2344 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "82ce84dd8a11441d88584a275c11e8c2" candidate_uuid: "ae60e0d407284144aa0861df39608d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00eb8e39eccd49f1ad6d989afb2fdd88" is_pre_election: true
I20261119 09:04:24.165640 2604 raft_consensus.cc:2490] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae60e0d407284144aa0861df39608d35 in term 0.
I20261119 09:04:24.165668 2344 raft_consensus.cc:2490] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae60e0d407284144aa0861df39608d35 in term 0.
I20261119 09:04:24.166105 2411 leader_election.cc:304] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9ef1f150cf7437eba4a78244f9cafb5, ae60e0d407284144aa0861df39608d35; no voters:
I20261119 09:04:24.166357 2677 raft_consensus.cc:2826] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261119 09:04:24.166431 2677 raft_consensus.cc:499] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261119 09:04:24.166476 2677 raft_consensus.cc:3096] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [term 0 FOLLOWER]: Advancing to term 1
W20261119 09:04:24.167155 2522 tablet.cc:2554] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261119 09:04:24.167364 2677 raft_consensus.cc:521] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00eb8e39eccd49f1ad6d989afb2fdd88" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 34869 } } peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } }
I20261119 09:04:24.167757 2677 leader_election.cc:290] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [CANDIDATE]: Term 1 election: Requested vote from peers 00eb8e39eccd49f1ad6d989afb2fdd88 (127.30.252.66:34869), a9ef1f150cf7437eba4a78244f9cafb5 (127.30.252.68:45669)
I20261119 09:04:24.168069 2344 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "82ce84dd8a11441d88584a275c11e8c2" candidate_uuid: "ae60e0d407284144aa0861df39608d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00eb8e39eccd49f1ad6d989afb2fdd88"
I20261119 09:04:24.168076 2604 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "82ce84dd8a11441d88584a275c11e8c2" candidate_uuid: "ae60e0d407284144aa0861df39608d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ef1f150cf7437eba4a78244f9cafb5"
I20261119 09:04:24.168181 2604 raft_consensus.cc:3096] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5 [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:24.168191 2344 raft_consensus.cc:3096] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88 [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:24.169018 2604 raft_consensus.cc:2490] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae60e0d407284144aa0861df39608d35 in term 1.
I20261119 09:04:24.169299 2411 leader_election.cc:304] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9ef1f150cf7437eba4a78244f9cafb5, ae60e0d407284144aa0861df39608d35; no voters:
I20261119 09:04:24.169456 2677 raft_consensus.cc:2826] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [term 1 FOLLOWER]: Leader election won for term 1
I20261119 09:04:24.169554 2344 raft_consensus.cc:2490] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae60e0d407284144aa0861df39608d35 in term 1.
I20261119 09:04:24.169723 2677 raft_consensus.cc:705] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [term 1 LEADER]: Becoming Leader. State: Replica: ae60e0d407284144aa0861df39608d35, State: Running, Role: LEADER
I20261119 09:04:24.170050 2677 consensus_queue.cc:238] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00eb8e39eccd49f1ad6d989afb2fdd88" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 34869 } } peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } }
I20261119 09:04:24.173359 2677 consensus_queue.cc:1049] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261119 09:04:24.173519 2681 consensus_queue.cc:1049] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00eb8e39eccd49f1ad6d989afb2fdd88" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 34869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261119 09:04:24.174115 2068 catalog_manager.cc:5856] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae60e0d407284144aa0861df39608d35 (127.30.252.67). New cstate: current_term: 1 leader_uuid: "ae60e0d407284144aa0861df39608d35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00eb8e39eccd49f1ad6d989afb2fdd88" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 34869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } health_report { overall_health: HEALTHY } } }
I20261119 09:04:24.200222 31729 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261119 09:04:24.202178 31729 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 00eb8e39eccd49f1ad6d989afb2fdd88 to finish bootstrapping
I20261119 09:04:24.208595 31729 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ae60e0d407284144aa0861df39608d35 to finish bootstrapping
I20261119 09:04:24.214017 31729 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a9ef1f150cf7437eba4a78244f9cafb5 to finish bootstrapping
I20261119 09:04:24.220850 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 2262
I20261119 09:04:24.245596 31729 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20261119 09:04:24.245890 31729 raft_consensus-itest-base.cc:268] Waiting until we've written at least 4MB...
W20261119 09:04:24.262159 2409 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.252.66:34869: connect: Connection refused (error 111)
W20261119 09:04:24.263355 2409 consensus_peers.cc:597] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 -> Peer 00eb8e39eccd49f1ad6d989afb2fdd88 (127.30.252.66:34869): Couldn't send request to peer 00eb8e39eccd49f1ad6d989afb2fdd88. Status: Network error: Client connection negotiation failed: client connection to 127.30.252.66:34869: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261119 09:04:24.331667 2653 maintenance_manager.cc:419] P a9ef1f150cf7437eba4a78244f9cafb5: Scheduling FlushMRSOp(82ce84dd8a11441d88584a275c11e8c2): 3681662 bytes log retention, and flush 5079305 bytes memory
I20261119 09:04:24.357062 31729 raft_consensus-itest-base.cc:274] Waiting for log GC on ae60e0d407284144aa0861df39608d35
I20261119 09:04:24.368727 2522 maintenance_manager.cc:419] P ae60e0d407284144aa0861df39608d35: Scheduling FlushMRSOp(82ce84dd8a11441d88584a275c11e8c2): 4602240 bytes log retention, and flush 6119689 bytes memory
I20261119 09:04:24.388862 2543 maintenance_manager.cc:643] P a9ef1f150cf7437eba4a78244f9cafb5: FlushMRSOp(82ce84dd8a11441d88584a275c11e8c2) complete. Timing: real 0.057s user 0.046s sys 0.004s Metrics: {"bytes_written":135937,"cfile_init":1,"compiler_manager_pool.queue_time_us":243,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":130,"dirs.run_wall_time_us":1216,"drs_written":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4,"lbm_write_time_us":662,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":29,"thread_start_us":121,"threads_started":1}
I20261119 09:04:24.390122 2653 maintenance_manager.cc:419] P a9ef1f150cf7437eba4a78244f9cafb5: Scheduling LogGCOp(82ce84dd8a11441d88584a275c11e8c2): free 2761276 bytes of WAL
I20261119 09:04:24.391817 2543 log_reader.cc:385] T 82ce84dd8a11441d88584a275c11e8c2: removed 3 log segments from log reader
I20261119 09:04:24.391952 2543 log.cc:1079] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal/wals/82ce84dd8a11441d88584a275c11e8c2/wal-000000001 (ops 1-8)
I20261119 09:04:24.392071 2543 log.cc:1079] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal/wals/82ce84dd8a11441d88584a275c11e8c2/wal-000000002 (ops 9-15)
I20261119 09:04:24.392143 2543 log.cc:1079] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal/wals/82ce84dd8a11441d88584a275c11e8c2/wal-000000003 (ops 16-22)
I20261119 09:04:24.393087 2543 maintenance_manager.cc:643] P a9ef1f150cf7437eba4a78244f9cafb5: LogGCOp(82ce84dd8a11441d88584a275c11e8c2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20261119 09:04:24.393432 2653 maintenance_manager.cc:419] P a9ef1f150cf7437eba4a78244f9cafb5: Scheduling UndoDeltaBlockGCOp(82ce84dd8a11441d88584a275c11e8c2): 460 bytes on disk
I20261119 09:04:24.394089 2543 maintenance_manager.cc:643] P a9ef1f150cf7437eba4a78244f9cafb5: UndoDeltaBlockGCOp(82ce84dd8a11441d88584a275c11e8c2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20261119 09:04:24.426313 2412 maintenance_manager.cc:643] P ae60e0d407284144aa0861df39608d35: FlushMRSOp(82ce84dd8a11441d88584a275c11e8c2) complete. Timing: real 0.057s user 0.052s sys 0.001s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":200,"dirs.queue_time_us":230,"dirs.run_cpu_time_us":250,"dirs.run_wall_time_us":1348,"drs_written":1,"lbm_read_time_us":284,"lbm_reads_lt_1ms":4,"lbm_write_time_us":508,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":38,"thread_start_us":215,"threads_started":2}
I20261119 09:04:24.427482 2522 maintenance_manager.cc:419] P ae60e0d407284144aa0861df39608d35: Scheduling LogGCOp(82ce84dd8a11441d88584a275c11e8c2): free 2761290 bytes of WAL
I20261119 09:04:24.427762 2412 log_reader.cc:385] T 82ce84dd8a11441d88584a275c11e8c2: removed 3 log segments from log reader
I20261119 09:04:24.427812 2412 log.cc:1079] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal/wals/82ce84dd8a11441d88584a275c11e8c2/wal-000000001 (ops 1-8)
I20261119 09:04:24.427865 2412 log.cc:1079] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal/wals/82ce84dd8a11441d88584a275c11e8c2/wal-000000002 (ops 9-15)
I20261119 09:04:24.427891 2412 log.cc:1079] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal/wals/82ce84dd8a11441d88584a275c11e8c2/wal-000000003 (ops 16-22)
I20261119 09:04:24.428783 2412 maintenance_manager.cc:643] P ae60e0d407284144aa0861df39608d35: LogGCOp(82ce84dd8a11441d88584a275c11e8c2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20261119 09:04:24.429111 2522 maintenance_manager.cc:419] P ae60e0d407284144aa0861df39608d35: Scheduling UndoDeltaBlockGCOp(82ce84dd8a11441d88584a275c11e8c2): 561 bytes on disk
I20261119 09:04:24.429421 31729 raft_consensus-itest-base.cc:279] Log GC complete on ae60e0d407284144aa0861df39608d35
I20261119 09:04:24.429876 2412 maintenance_manager.cc:643] P ae60e0d407284144aa0861df39608d35: UndoDeltaBlockGCOp(82ce84dd8a11441d88584a275c11e8c2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20261119 09:04:24.499234 2259 heartbeater.cc:500] Master 127.30.252.126:40941 was elected leader, sending a full tablet report...
I20261119 09:04:24.767467 2706 consensus_queue.cc:754] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [LEADER]: The logs necessary to catch up peer 00eb8e39eccd49f1ad6d989afb2fdd88 have been garbage collected. The follower will never be able to catch up (Not found: failed to read ops 3..33: Segment 1 which contained index 3 has been GCed)
I20261119 09:04:24.767609 2706 consensus_queue.cc:580] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [LEADER]: The logs necessary to catch up peer 00eb8e39eccd49f1ad6d989afb2fdd88 have been garbage collected. The replica will never be able to catch up
I20261119 09:04:24.770994 2475 consensus_queue.cc:238] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } }
I20261119 09:04:24.772415 2606 raft_consensus.cc:1285] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5 [term 1 FOLLOWER]: Refusing update from remote peer ae60e0d407284144aa0861df39608d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 40. (index mismatch)
I20261119 09:04:24.772891 2679 consensus_queue.cc:1049] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20261119 09:04:24.774943 2706 raft_consensus.cc:2983] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [term 1 LEADER]: Committing config change with OpId 1.40: config changed from index -1 to 40, VOTER 00eb8e39eccd49f1ad6d989afb2fdd88 (127.30.252.66) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } } }
I20261119 09:04:24.775415 2606 raft_consensus.cc:2983] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5 [term 1 FOLLOWER]: Committing config change with OpId 1.40: config changed from index -1 to 40, VOTER 00eb8e39eccd49f1ad6d989afb2fdd88 (127.30.252.66) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } } }
I20261119 09:04:24.776693 2054 catalog_manager.cc:5263] ChangeConfig:REMOVE_PEER RPC for tablet 82ce84dd8a11441d88584a275c11e8c2 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20261119 09:04:24.777666 2068 catalog_manager.cc:5856] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 reported cstate change: config changed from index -1 to 40, VOTER 00eb8e39eccd49f1ad6d989afb2fdd88 (127.30.252.66) evicted. New cstate: current_term: 1 leader_uuid: "ae60e0d407284144aa0861df39608d35" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } health_report { overall_health: HEALTHY } } }
W20261119 09:04:24.781922 2055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.252.66:34869: connect: Connection refused (error 111)
I20261119 09:04:24.782706 2475 consensus_queue.cc:238] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } } peers { permanent_uuid: "3e06e721e65e492891128523e5df549d" member_type: NON_VOTER last_known_addr { host: "127.30.252.65" port: 46315 } attrs { promote: true } }
W20261119 09:04:24.783039 2055 catalog_manager.cc:4808] TS 00eb8e39eccd49f1ad6d989afb2fdd88 (127.30.252.66:34869): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 82ce84dd8a11441d88584a275c11e8c2: Network error: Client connection negotiation failed: client connection to 127.30.252.66:34869: connect: Connection refused (error 111)
I20261119 09:04:24.784633 2606 raft_consensus.cc:1285] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5 [term 1 FOLLOWER]: Refusing update from remote peer ae60e0d407284144aa0861df39608d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 1 index: 41. (index mismatch)
I20261119 09:04:24.785204 2679 consensus_queue.cc:1049] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20261119 09:04:24.787205 2706 raft_consensus.cc:2983] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [term 1 LEADER]: Committing config change with OpId 1.41: config changed from index 40 to 41, NON_VOTER 3e06e721e65e492891128523e5df549d (127.30.252.65) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } } peers { permanent_uuid: "3e06e721e65e492891128523e5df549d" member_type: NON_VOTER last_known_addr { host: "127.30.252.65" port: 46315 } attrs { promote: true } } }
I20261119 09:04:24.787803 2606 raft_consensus.cc:2983] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5 [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index 40 to 41, NON_VOTER 3e06e721e65e492891128523e5df549d (127.30.252.65) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } } peers { permanent_uuid: "3e06e721e65e492891128523e5df549d" member_type: NON_VOTER last_known_addr { host: "127.30.252.65" port: 46315 } attrs { promote: true } } }
I20261119 09:04:24.788697 2054 catalog_manager.cc:5263] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 82ce84dd8a11441d88584a275c11e8c2 with cas_config_opid_index 40: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20261119 09:04:24.790011 2067 catalog_manager.cc:5856] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 reported cstate change: config changed from index 40 to 41, NON_VOTER 3e06e721e65e492891128523e5df549d (127.30.252.65) added. New cstate: current_term: 1 leader_uuid: "ae60e0d407284144aa0861df39608d35" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e06e721e65e492891128523e5df549d" member_type: NON_VOTER last_known_addr { host: "127.30.252.65" port: 46315 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20261119 09:04:24.791656 2410 consensus_peers.cc:597] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 -> Peer 3e06e721e65e492891128523e5df549d (127.30.252.65:46315): Couldn't send request to peer 3e06e721e65e492891128523e5df549d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82ce84dd8a11441d88584a275c11e8c2. This is attempt 1: this message will repeat every 5th retry.
I20261119 09:04:25.319408 2719 ts_tablet_manager.cc:979] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d: Initiating tablet copy from peer ae60e0d407284144aa0861df39608d35 (127.30.252.67:44273)
I20261119 09:04:25.320060 2719 tablet_copy_client.cc:323] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d: tablet copy: Beginning tablet copy session from remote peer at address 127.30.252.67:44273
I20261119 09:04:25.324038 2495 tablet_copy_service.cc:140] P ae60e0d407284144aa0861df39608d35: Received BeginTabletCopySession request for tablet 82ce84dd8a11441d88584a275c11e8c2 from peer 3e06e721e65e492891128523e5df549d ({username='slave'} at 127.30.252.65:52249)
I20261119 09:04:25.324174 2495 tablet_copy_service.cc:161] P ae60e0d407284144aa0861df39608d35: Beginning new tablet copy session on tablet 82ce84dd8a11441d88584a275c11e8c2 from peer 3e06e721e65e492891128523e5df549d at {username='slave'} at 127.30.252.65:52249: session id = 3e06e721e65e492891128523e5df549d-82ce84dd8a11441d88584a275c11e8c2
I20261119 09:04:25.325721 2495 tablet_copy_source_session.cc:215] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35: Tablet Copy: opened 5 blocks and 3 log segments
I20261119 09:04:25.329936 2719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82ce84dd8a11441d88584a275c11e8c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:25.334939 2719 tablet_copy_client.cc:806] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d: tablet copy: Starting download of 5 data blocks...
I20261119 09:04:25.343883 2719 tablet_copy_client.cc:670] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d: tablet copy: Starting download of 3 WAL segments...
I20261119 09:04:25.352061 2719 tablet_copy_client.cc:538] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261119 09:04:25.354665 2719 tablet_bootstrap.cc:492] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d: Bootstrap starting.
I20261119 09:04:25.363494 2719 log.cc:826] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:25.370951 2719 tablet_bootstrap.cc:492] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=6 ignored=6} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=3. Pending: 1 replicates
I20261119 09:04:25.374987 2719 tablet_bootstrap.cc:492] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=12 ignored=12} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=3. Pending: 2 replicates
I20261119 09:04:25.378088 2719 tablet_bootstrap.cc:492] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d: Bootstrap replayed 3/3 log segments. Stats: ops{read=19 overwritten=0 applied=19 ignored=17} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=3. Pending: 0 replicates
I20261119 09:04:25.378391 2719 tablet_bootstrap.cc:492] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d: Bootstrap complete.
I20261119 09:04:25.379033 2719 ts_tablet_manager.cc:1449] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d: Time spent bootstrapping tablet: real 0.025s user 0.017s sys 0.007s
I20261119 09:04:25.381062 2719 raft_consensus.cc:360] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } } peers { permanent_uuid: "3e06e721e65e492891128523e5df549d" member_type: NON_VOTER last_known_addr { host: "127.30.252.65" port: 46315 } attrs { promote: true } }
I20261119 09:04:25.381278 2719 raft_consensus.cc:748] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e06e721e65e492891128523e5df549d, State: Initialized, Role: LEARNER
I20261119 09:04:25.381837 2719 consensus_queue.cc:261] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 41, Last appended: 1.41, Last appended by leader: 41, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } } peers { permanent_uuid: "3e06e721e65e492891128523e5df549d" member_type: NON_VOTER last_known_addr { host: "127.30.252.65" port: 46315 } attrs { promote: true } }
I20261119 09:04:25.382567 2719 ts_tablet_manager.cc:1480] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d: Time spent starting tablet: real 0.003s user 0.006s sys 0.001s
I20261119 09:04:25.383497 2494 tablet_copy_service.cc:342] P ae60e0d407284144aa0861df39608d35: Request end of tablet copy session 3e06e721e65e492891128523e5df549d-82ce84dd8a11441d88584a275c11e8c2 received from {username='slave'} at 127.30.252.65:52249
I20261119 09:04:25.383641 2494 tablet_copy_service.cc:434] P ae60e0d407284144aa0861df39608d35: ending tablet copy session 3e06e721e65e492891128523e5df549d-82ce84dd8a11441d88584a275c11e8c2 on tablet 82ce84dd8a11441d88584a275c11e8c2 with peer 3e06e721e65e492891128523e5df549d
I20261119 09:04:25.389678 2260 maintenance_manager.cc:419] P 3e06e721e65e492891128523e5df549d: Scheduling UndoDeltaBlockGCOp(82ce84dd8a11441d88584a275c11e8c2): 561 bytes on disk
I20261119 09:04:25.390463 2149 maintenance_manager.cc:643] P 3e06e721e65e492891128523e5df549d: UndoDeltaBlockGCOp(82ce84dd8a11441d88584a275c11e8c2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
I20261119 09:04:25.764331 2213 raft_consensus.cc:1227] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.40->[1.41-1.41] Dedup: 1.41->[]
I20261119 09:04:25.836059 2522 maintenance_manager.cc:419] P ae60e0d407284144aa0861df39608d35: Scheduling LogGCOp(82ce84dd8a11441d88584a275c11e8c2): free 1840950 bytes of WAL
I20261119 09:04:25.836313 2412 log_reader.cc:385] T 82ce84dd8a11441d88584a275c11e8c2: removed 2 log segments from log reader
I20261119 09:04:25.836355 2412 log.cc:1079] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal/wals/82ce84dd8a11441d88584a275c11e8c2/wal-000000004 (ops 23-29)
I20261119 09:04:25.836391 2412 log.cc:1079] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal/wals/82ce84dd8a11441d88584a275c11e8c2/wal-000000005 (ops 30-36)
I20261119 09:04:25.837037 2412 maintenance_manager.cc:643] P ae60e0d407284144aa0861df39608d35: LogGCOp(82ce84dd8a11441d88584a275c11e8c2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
W20261119 09:04:25.995798 2055 catalog_manager.cc:4808] TS 00eb8e39eccd49f1ad6d989afb2fdd88 (127.30.252.66:34869): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 82ce84dd8a11441d88584a275c11e8c2: Network error: Client connection negotiation failed: client connection to 127.30.252.66:34869: connect: Connection refused (error 111)
I20261119 09:04:26.177779 2727 raft_consensus.cc:1074] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35: attempting to promote NON_VOTER 3e06e721e65e492891128523e5df549d to VOTER
I20261119 09:04:26.178268 2727 consensus_queue.cc:238] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 1.41, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } } peers { permanent_uuid: "3e06e721e65e492891128523e5df549d" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 46315 } attrs { promote: false } }
I20261119 09:04:26.179625 2606 raft_consensus.cc:1285] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5 [term 1 FOLLOWER]: Refusing update from remote peer ae60e0d407284144aa0861df39608d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 41. Preceding OpId from leader: term: 1 index: 42. (index mismatch)
I20261119 09:04:26.179734 2213 raft_consensus.cc:1285] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d [term 1 LEARNER]: Refusing update from remote peer ae60e0d407284144aa0861df39608d35: Log matching property violated. Preceding OpId in replica: term: 1 index: 41. Preceding OpId from leader: term: 1 index: 42. (index mismatch)
I20261119 09:04:26.180023 2728 consensus_queue.cc:1049] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20261119 09:04:26.180203 2727 consensus_queue.cc:1049] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e06e721e65e492891128523e5df549d" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 46315 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20261119 09:04:26.182189 2727 raft_consensus.cc:2983] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 [term 1 LEADER]: Committing config change with OpId 1.42: config changed from index 41 to 42, 3e06e721e65e492891128523e5df549d (127.30.252.65) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } } peers { permanent_uuid: "3e06e721e65e492891128523e5df549d" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 46315 } attrs { promote: false } } }
I20261119 09:04:26.182551 2606 raft_consensus.cc:2983] T 82ce84dd8a11441d88584a275c11e8c2 P a9ef1f150cf7437eba4a78244f9cafb5 [term 1 FOLLOWER]: Committing config change with OpId 1.42: config changed from index 41 to 42, 3e06e721e65e492891128523e5df549d (127.30.252.65) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } } peers { permanent_uuid: "3e06e721e65e492891128523e5df549d" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 46315 } attrs { promote: false } } }
I20261119 09:04:26.184516 2213 raft_consensus.cc:2983] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d [term 1 FOLLOWER]: Committing config change with OpId 1.42: config changed from index 41 to 42, 3e06e721e65e492891128523e5df549d (127.30.252.65) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } } peers { permanent_uuid: "3e06e721e65e492891128523e5df549d" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 46315 } attrs { promote: false } } }
I20261119 09:04:26.184675 2068 catalog_manager.cc:5856] T 82ce84dd8a11441d88584a275c11e8c2 P ae60e0d407284144aa0861df39608d35 reported cstate change: config changed from index 41 to 42, 3e06e721e65e492891128523e5df549d (127.30.252.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ae60e0d407284144aa0861df39608d35" committed_config { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e06e721e65e492891128523e5df549d" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 46315 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20261119 09:04:26.194547 2260 maintenance_manager.cc:419] P 3e06e721e65e492891128523e5df549d: Scheduling LogGCOp(82ce84dd8a11441d88584a275c11e8c2): free 1840860 bytes of WAL
I20261119 09:04:26.194865 2149 log_reader.cc:385] T 82ce84dd8a11441d88584a275c11e8c2: removed 2 log segments from log reader
I20261119 09:04:26.194928 2149 log.cc:1079] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal/wals/82ce84dd8a11441d88584a275c11e8c2/wal-000000001 (ops 23-29)
I20261119 09:04:26.194979 2149 log.cc:1079] T 82ce84dd8a11441d88584a275c11e8c2 P 3e06e721e65e492891128523e5df549d: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal/wals/82ce84dd8a11441d88584a275c11e8c2/wal-000000002 (ops 30-36)
I20261119 09:04:26.195466 2149 maintenance_manager.cc:643] P 3e06e721e65e492891128523e5df549d: LogGCOp(82ce84dd8a11441d88584a275c11e8c2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20261119 09:04:26.430742 31729 raft_consensus-itest-base.cc:295] Servers converged with original term 1
I20261119 09:04:26.430809 31729 raft_consensus-itest-base.cc:303] Restarting 00eb8e39eccd49f1ad6d989afb2fdd88
I20261119 09:04:26.431088 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.66:34869
--local_ip_for_outbound_sockets=127.30.252.66
--tserver_master_addrs=127.30.252.126:40941
--webserver_port=36029
--webserver_interface=127.30.252.66
--builtin_ntp_servers=127.30.252.84:45905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20261119 09:04:26.532954 2740 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261119 09:04:26.533164 2740 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:26.533188 2740 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:26.533254 2740 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:26.536729 2740 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261119 09:04:26.536791 2740 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261119 09:04:26.536826 2740 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:26.536907 2740 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261119 09:04:26.536950 2740 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.66
I20261119 09:04:26.540948 2740 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:45905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.66:34869
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.252.66
--webserver_port=36029
--tserver_master_addrs=127.30.252.126:40941
--never_fsync=true
--heap_profile_path=/tmp/kudu.2740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:26.541740 2740 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:26.542441 2740 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:26.548548 2748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:26.548736 2746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:26.548760 2745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:26.549002 2740 server_base.cc:1061] running on GCE node
I20261119 09:04:26.549316 2740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:26.549867 2740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:26.551044 2740 hybrid_clock.cc:648] HybridClock initialized: now 1795079066551023 us; error 37 us; skew 500 ppm
I20261119 09:04:26.552980 2740 webserver.cc:577] Webserver started at http://127.30.252.66:36029/ using document root <none> and password file <none>
I20261119 09:04:26.553570 2740 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:26.553632 2740 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:26.557065 2740 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20261119 09:04:26.559125 2754 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:26.560171 2740 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261119 09:04:26.560310 2740 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "00eb8e39eccd49f1ad6d989afb2fdd88"
format_stamp: "Formatted at 2026-11-19 09:04:23 on dist-test-slave-1fcg"
I20261119 09:04:26.560678 2740 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:26.570747 2740 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:26.571179 2740 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:26.571364 2740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:26.571894 2740 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:26.573055 2761 ts_tablet_manager.cc:588] Loading tablet metadata (0/1 complete)
I20261119 09:04:26.576556 2740 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261119 09:04:26.576622 2740 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20261119 09:04:26.576689 2740 ts_tablet_manager.cc:646] Registering tablets (0/1 complete)
I20261119 09:04:26.578183 2740 ts_tablet_manager.cc:662] Registered 1 tablets
I20261119 09:04:26.578225 2740 ts_tablet_manager.cc:641] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20261119 09:04:26.578589 2761 tablet_bootstrap.cc:492] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88: Bootstrap starting.
I20261119 09:04:26.585438 2761 log.cc:826] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:26.587591 2761 tablet_bootstrap.cc:492] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261119 09:04:26.587790 2740 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.66:34869
I20261119 09:04:26.587972 2869 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.66:34869 every 8 connection(s)
I20261119 09:04:26.588059 2761 tablet_bootstrap.cc:492] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88: Bootstrap complete.
I20261119 09:04:26.588421 2761 ts_tablet_manager.cc:1449] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20261119 09:04:26.588734 2740 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20261119 09:04:26.591517 2761 raft_consensus.cc:360] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00eb8e39eccd49f1ad6d989afb2fdd88" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 34869 } } peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } }
I20261119 09:04:26.591917 2761 raft_consensus.cc:748] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00eb8e39eccd49f1ad6d989afb2fdd88, State: Initialized, Role: FOLLOWER
I20261119 09:04:26.592372 2761 consensus_queue.cc:261] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00eb8e39eccd49f1ad6d989afb2fdd88" member_type: VOTER last_known_addr { host: "127.30.252.66" port: 34869 } } peers { permanent_uuid: "a9ef1f150cf7437eba4a78244f9cafb5" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 45669 } } peers { permanent_uuid: "ae60e0d407284144aa0861df39608d35" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 44273 } }
I20261119 09:04:26.593067 2761 ts_tablet_manager.cc:1480] T 82ce84dd8a11441d88584a275c11e8c2 P 00eb8e39eccd49f1ad6d989afb2fdd88: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20261119 09:04:26.596319 2870 heartbeater.cc:345] Connected to a master server at 127.30.252.126:40941
I20261119 09:04:26.596526 2870 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:26.597033 2870 heartbeater.cc:508] Master 127.30.252.126:40941 requested a full tablet report, sending...
I20261119 09:04:26.597055 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 2740
I20261119 09:04:26.599119 2069 ts_manager.cc:210] Re-registered known tserver with Master: 00eb8e39eccd49f1ad6d989afb2fdd88 (127.30.252.66:34869)
I20261119 09:04:26.600049 2069 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.66:54169
I20261119 09:04:26.601454 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 2130
I20261119 09:04:26.604578 2870 heartbeater.cc:500] Master 127.30.252.126:40941 was elected leader, sending a full tablet report...
I20261119 09:04:26.608922 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 2740
I20261119 09:04:26.614368 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 2393
I20261119 09:04:26.620504 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 2524
I20261119 09:04:26.626518 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 2039
2026-11-19T09:04:26Z chronyd exiting
[ OK ] SlowTabletBootstrapTest.TestFallBehind (3580 ms)
[ RUN ] SlowTabletBootstrapTest.TestFallBehindSlowBootstrap
I20261119 09:04:26.649690 31729 ts_itest-base.cc:115] Starting cluster with:
I20261119 09:04:26.649735 31729 ts_itest-base.cc:116] --------------
I20261119 09:04:26.649745 31729 ts_itest-base.cc:117] 4 tablet servers
I20261119 09:04:26.649755 31729 ts_itest-base.cc:118] 3 replicas per TS
I20261119 09:04:26.649763 31729 ts_itest-base.cc:119] --------------
2026-11-19T09:04:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
Could not open connection to daemon
2026-11-19T09:04:26Z Disabled control of system clock
I20261119 09:04:26.661979 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.252.126:35689
--webserver_interface=127.30.252.126
--webserver_port=0
--builtin_ntp_servers=127.30.252.84:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.252.126:35689 with env {}
W20261119 09:04:26.771795 2883 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:26.771975 2883 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:26.772034 2883 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:26.775483 2883 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261119 09:04:26.775552 2883 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:26.775610 2883 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261119 09:04:26.775640 2883 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261119 09:04:26.779765 2883 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.252.126:35689
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.252.126:35689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.252.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:26.780613 2883 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:26.781306 2883 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:26.787209 2891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:26.787200 2888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:26.787197 2889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:26.787508 2883 server_base.cc:1061] running on GCE node
I20261119 09:04:26.787950 2883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:26.788643 2883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:26.789803 2883 hybrid_clock.cc:648] HybridClock initialized: now 1795079066789781 us; error 32 us; skew 500 ppm
I20261119 09:04:26.791417 2883 webserver.cc:577] Webserver started at http://127.30.252.126:36447/ using document root <none> and password file <none>
I20261119 09:04:26.792001 2883 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:26.792052 2883 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:26.792228 2883 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:26.793480 2883 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1d78c24e04294aba996b4d6874a2dbef"
format_stamp: "Formatted at 2026-11-19 09:04:26 on dist-test-slave-1fcg"
I20261119 09:04:26.793848 2883 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1d78c24e04294aba996b4d6874a2dbef"
format_stamp: "Formatted at 2026-11-19 09:04:26 on dist-test-slave-1fcg"
I20261119 09:04:26.797001 2883 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20261119 09:04:26.799129 2897 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:26.800205 2883 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261119 09:04:26.800362 2883 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1d78c24e04294aba996b4d6874a2dbef"
format_stamp: "Formatted at 2026-11-19 09:04:26 on dist-test-slave-1fcg"
I20261119 09:04:26.800458 2883 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:26.814162 2883 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:26.814674 2883 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:26.814841 2883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:26.821839 2883 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.126:35689
I20261119 09:04:26.821851 2949 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.126:35689 every 8 connection(s)
I20261119 09:04:26.822819 2883 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20261119 09:04:26.825915 2950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:26.827386 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 2883
I20261119 09:04:26.827615 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/master-0/wal/instance
I20261119 09:04:26.831529 2950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef: Bootstrap starting.
I20261119 09:04:26.834052 2950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:26.834918 2950 log.cc:826] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:26.836284 2950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef: No bootstrap required, opened a new log
I20261119 09:04:26.839912 2950 raft_consensus.cc:360] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d78c24e04294aba996b4d6874a2dbef" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 35689 } }
I20261119 09:04:26.840157 2950 raft_consensus.cc:386] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:04:26.840225 2950 raft_consensus.cc:748] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d78c24e04294aba996b4d6874a2dbef, State: Initialized, Role: FOLLOWER
I20261119 09:04:26.840724 2950 consensus_queue.cc:261] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d78c24e04294aba996b4d6874a2dbef" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 35689 } }
I20261119 09:04:26.840894 2950 raft_consensus.cc:400] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261119 09:04:26.840962 2950 raft_consensus.cc:499] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261119 09:04:26.841054 2950 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:26.841872 2950 raft_consensus.cc:521] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d78c24e04294aba996b4d6874a2dbef" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 35689 } }
I20261119 09:04:26.842550 2950 leader_election.cc:304] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d78c24e04294aba996b4d6874a2dbef; no voters:
I20261119 09:04:26.842856 2950 leader_election.cc:290] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef [CANDIDATE]: Term 1 election: Requested vote from peers
I20261119 09:04:26.842944 2955 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef [term 1 FOLLOWER]: Leader election won for term 1
I20261119 09:04:26.843133 2955 raft_consensus.cc:705] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef [term 1 LEADER]: Becoming Leader. State: Replica: 1d78c24e04294aba996b4d6874a2dbef, State: Running, Role: LEADER
I20261119 09:04:26.843499 2955 consensus_queue.cc:238] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d78c24e04294aba996b4d6874a2dbef" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 35689 } }
I20261119 09:04:26.844058 2950 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef [sys.catalog]: configured and running, proceeding with master startup.
I20261119 09:04:26.845160 2956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d78c24e04294aba996b4d6874a2dbef. Latest consensus state: current_term: 1 leader_uuid: "1d78c24e04294aba996b4d6874a2dbef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d78c24e04294aba996b4d6874a2dbef" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 35689 } } }
I20261119 09:04:26.845448 2957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d78c24e04294aba996b4d6874a2dbef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d78c24e04294aba996b4d6874a2dbef" member_type: VOTER last_known_addr { host: "127.30.252.126" port: 35689 } } }
I20261119 09:04:26.845556 2956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef [sys.catalog]: This master's current role is: LEADER
I20261119 09:04:26.845747 2957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef [sys.catalog]: This master's current role is: LEADER
W20261119 09:04:26.848687 2971 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20261119 09:04:26.848747 2971 catalog_manager.cc:898] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20261119 09:04:26.848819 2967 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261119 09:04:26.849607 2967 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261119 09:04:26.854691 2967 catalog_manager.cc:1398] Generated new cluster ID: 747a3d51346f4fef85b40b00eebb6849
I20261119 09:04:26.854779 2967 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261119 09:04:26.865162 2967 catalog_manager.cc:1421] Generated new certificate authority record
I20261119 09:04:26.866346 2967 catalog_manager.cc:1555] Loading token signing keys...
I20261119 09:04:26.872740 2967 catalog_manager.cc:6237] T 00000000000000000000000000000000 P 1d78c24e04294aba996b4d6874a2dbef: Generated new TSK 0
I20261119 09:04:26.873559 2967 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261119 09:04:26.876884 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.65:0
--local_ip_for_outbound_sockets=127.30.252.65
--webserver_interface=127.30.252.65
--webserver_port=0
--tserver_master_addrs=127.30.252.126:35689
--builtin_ntp_servers=127.30.252.84:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20261119 09:04:26.982303 2974 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261119 09:04:26.982553 2974 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:26.982599 2974 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:26.982666 2974 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:26.986328 2974 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261119 09:04:26.986387 2974 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261119 09:04:26.986443 2974 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:26.986547 2974 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261119 09:04:26.986585 2974 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.65
I20261119 09:04:26.990715 2974 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.252.65
--webserver_port=0
--tserver_master_addrs=127.30.252.126:35689
--never_fsync=true
--heap_profile_path=/tmp/kudu.2974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:26.991523 2974 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:26.992254 2974 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:26.998327 2982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:26.998486 2974 server_base.cc:1061] running on GCE node
W20261119 09:04:26.998481 2979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:26.998327 2980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:26.999017 2974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:26.999608 2974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:27.000790 2974 hybrid_clock.cc:648] HybridClock initialized: now 1795079067000760 us; error 47 us; skew 500 ppm
I20261119 09:04:27.002764 2974 webserver.cc:577] Webserver started at http://127.30.252.65:39879/ using document root <none> and password file <none>
I20261119 09:04:27.003372 2974 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:27.003423 2974 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:27.003808 2974 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:27.006086 2974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d774fe05a5d048fd9b410188ad7302b8"
format_stamp: "Formatted at 2026-11-19 09:04:27 on dist-test-slave-1fcg"
I20261119 09:04:27.006699 2974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d774fe05a5d048fd9b410188ad7302b8"
format_stamp: "Formatted at 2026-11-19 09:04:27 on dist-test-slave-1fcg"
I20261119 09:04:27.010248 2974 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261119 09:04:27.012602 2988 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:27.013554 2974 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261119 09:04:27.013653 2974 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d774fe05a5d048fd9b410188ad7302b8"
format_stamp: "Formatted at 2026-11-19 09:04:27 on dist-test-slave-1fcg"
I20261119 09:04:27.013744 2974 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:27.038676 2974 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:27.039163 2974 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:27.039309 2974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:27.039846 2974 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:27.040791 2974 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:27.040841 2974 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:27.040884 2974 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:27.040897 2974 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:27.050061 2974 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.65:34525
I20261119 09:04:27.050161 3101 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.65:34525 every 8 connection(s)
I20261119 09:04:27.051084 2974 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20261119 09:04:27.053174 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 2974
I20261119 09:04:27.053332 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20261119 09:04:27.055596 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.66:0
--local_ip_for_outbound_sockets=127.30.252.66
--webserver_interface=127.30.252.66
--webserver_port=0
--tserver_master_addrs=127.30.252.126:35689
--builtin_ntp_servers=127.30.252.84:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261119 09:04:27.059725 3102 heartbeater.cc:345] Connected to a master server at 127.30.252.126:35689
I20261119 09:04:27.059943 3102 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:27.060425 3102 heartbeater.cc:508] Master 127.30.252.126:35689 requested a full tablet report, sending...
I20261119 09:04:27.061995 2914 ts_manager.cc:210] Registered new tserver with Master: d774fe05a5d048fd9b410188ad7302b8 (127.30.252.65:34525)
I20261119 09:04:27.063235 2914 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.65:48333
W20261119 09:04:27.155861 3105 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261119 09:04:27.156098 3105 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:27.156140 3105 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:27.156207 3105 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:27.159664 3105 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261119 09:04:27.159718 3105 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261119 09:04:27.159768 3105 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:27.159864 3105 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261119 09:04:27.159901 3105 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.66
I20261119 09:04:27.163959 3105 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.252.66
--webserver_port=0
--tserver_master_addrs=127.30.252.126:35689
--never_fsync=true
--heap_profile_path=/tmp/kudu.3105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:27.164748 3105 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:27.165462 3105 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:27.171078 3111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:27.171136 3113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:27.171231 3105 server_base.cc:1061] running on GCE node
W20261119 09:04:27.171142 3110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:27.171720 3105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:27.172259 3105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:27.173411 3105 hybrid_clock.cc:648] HybridClock initialized: now 1795079067173412 us; error 43 us; skew 500 ppm
I20261119 09:04:27.175103 3105 webserver.cc:577] Webserver started at http://127.30.252.66:37133/ using document root <none> and password file <none>
I20261119 09:04:27.175686 3105 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:27.175743 3105 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:27.175959 3105 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:27.177202 3105 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "224f3894e1544e7ea41b5509167e9271"
format_stamp: "Formatted at 2026-11-19 09:04:27 on dist-test-slave-1fcg"
I20261119 09:04:27.177632 3105 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "224f3894e1544e7ea41b5509167e9271"
format_stamp: "Formatted at 2026-11-19 09:04:27 on dist-test-slave-1fcg"
I20261119 09:04:27.180765 3105 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20261119 09:04:27.183077 3119 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:27.184151 3105 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261119 09:04:27.184268 3105 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "224f3894e1544e7ea41b5509167e9271"
format_stamp: "Formatted at 2026-11-19 09:04:27 on dist-test-slave-1fcg"
I20261119 09:04:27.184353 3105 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:27.199988 3105 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:27.200417 3105 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:27.200549 3105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:27.201041 3105 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:27.201977 3105 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:27.202023 3105 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:27.202077 3105 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:27.202106 3105 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:27.210736 3105 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.66:41595
I20261119 09:04:27.210783 3233 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.66:41595 every 8 connection(s)
I20261119 09:04:27.211736 3105 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20261119 09:04:27.219748 3234 heartbeater.cc:345] Connected to a master server at 127.30.252.126:35689
I20261119 09:04:27.219998 3234 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:27.220480 3234 heartbeater.cc:508] Master 127.30.252.126:35689 requested a full tablet report, sending...
I20261119 09:04:27.221643 2914 ts_manager.cc:210] Registered new tserver with Master: 224f3894e1544e7ea41b5509167e9271 (127.30.252.66:41595)
I20261119 09:04:27.221944 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 3105
I20261119 09:04:27.222041 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20261119 09:04:27.222291 2914 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.66:45251
I20261119 09:04:27.224303 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.67:0
--local_ip_for_outbound_sockets=127.30.252.67
--webserver_interface=127.30.252.67
--webserver_port=0
--tserver_master_addrs=127.30.252.126:35689
--builtin_ntp_servers=127.30.252.84:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20261119 09:04:27.328578 3237 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261119 09:04:27.328783 3237 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:27.328814 3237 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:27.328910 3237 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:27.332505 3237 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261119 09:04:27.332566 3237 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261119 09:04:27.332602 3237 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:27.332666 3237 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261119 09:04:27.332696 3237 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.67
I20261119 09:04:27.336776 3237 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.252.67
--webserver_port=0
--tserver_master_addrs=127.30.252.126:35689
--never_fsync=true
--heap_profile_path=/tmp/kudu.3237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:27.337522 3237 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:27.338174 3237 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:27.343912 3242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:27.343905 3245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:27.344075 3237 server_base.cc:1061] running on GCE node
W20261119 09:04:27.344103 3243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:27.344519 3237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:27.345162 3237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:27.346316 3237 hybrid_clock.cc:648] HybridClock initialized: now 1795079067346299 us; error 33 us; skew 500 ppm
I20261119 09:04:27.347952 3237 webserver.cc:577] Webserver started at http://127.30.252.67:42527/ using document root <none> and password file <none>
I20261119 09:04:27.348513 3237 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:27.348563 3237 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:27.348759 3237 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:27.350054 3237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e"
format_stamp: "Formatted at 2026-11-19 09:04:27 on dist-test-slave-1fcg"
I20261119 09:04:27.350482 3237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e"
format_stamp: "Formatted at 2026-11-19 09:04:27 on dist-test-slave-1fcg"
I20261119 09:04:27.353633 3237 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261119 09:04:27.356051 3251 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:27.356979 3237 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261119 09:04:27.357096 3237 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e"
format_stamp: "Formatted at 2026-11-19 09:04:27 on dist-test-slave-1fcg"
I20261119 09:04:27.357179 3237 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:27.395257 3237 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:27.395741 3237 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:27.395884 3237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:27.396394 3237 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:27.397431 3237 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:27.397487 3237 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:27.397540 3237 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:27.397562 3237 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:27.405725 3237 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.67:42671
I20261119 09:04:27.405761 3364 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.67:42671 every 8 connection(s)
I20261119 09:04:27.406646 3237 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20261119 09:04:27.410266 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 3237
I20261119 09:04:27.410393 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20261119 09:04:27.412689 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.68:0
--local_ip_for_outbound_sockets=127.30.252.68
--webserver_interface=127.30.252.68
--webserver_port=0
--tserver_master_addrs=127.30.252.126:35689
--builtin_ntp_servers=127.30.252.84:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261119 09:04:27.414340 3365 heartbeater.cc:345] Connected to a master server at 127.30.252.126:35689
I20261119 09:04:27.414569 3365 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:27.415046 3365 heartbeater.cc:508] Master 127.30.252.126:35689 requested a full tablet report, sending...
I20261119 09:04:27.416138 2914 ts_manager.cc:210] Registered new tserver with Master: 8c7f7565a51d4e2f9b80bb3aff51fd8e (127.30.252.67:42671)
I20261119 09:04:27.416875 2914 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.67:48285
W20261119 09:04:27.519361 3368 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261119 09:04:27.519618 3368 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:27.519663 3368 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:27.519725 3368 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:27.523238 3368 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261119 09:04:27.523300 3368 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261119 09:04:27.523350 3368 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:27.523440 3368 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261119 09:04:27.523479 3368 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.68
I20261119 09:04:27.527662 3368 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.30.252.68
--webserver_port=0
--tserver_master_addrs=127.30.252.126:35689
--never_fsync=true
--heap_profile_path=/tmp/kudu.3368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.68
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:27.528501 3368 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:27.529347 3368 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:27.535436 3376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:27.535445 3374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:27.535591 3368 server_base.cc:1061] running on GCE node
W20261119 09:04:27.535437 3373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:27.536072 3368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:27.536619 3368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:27.537796 3368 hybrid_clock.cc:648] HybridClock initialized: now 1795079067537780 us; error 28 us; skew 500 ppm
I20261119 09:04:27.539439 3368 webserver.cc:577] Webserver started at http://127.30.252.68:45875/ using document root <none> and password file <none>
I20261119 09:04:27.540015 3368 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:27.540067 3368 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:27.540271 3368 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261119 09:04:27.541517 3368 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1a51220b262b4a93b5e26e30bb3e8b5a"
format_stamp: "Formatted at 2026-11-19 09:04:27 on dist-test-slave-1fcg"
I20261119 09:04:27.541932 3368 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1a51220b262b4a93b5e26e30bb3e8b5a"
format_stamp: "Formatted at 2026-11-19 09:04:27 on dist-test-slave-1fcg"
I20261119 09:04:27.546177 3368 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20261119 09:04:27.548714 3382 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:27.549901 3368 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20261119 09:04:27.550012 3368 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1a51220b262b4a93b5e26e30bb3e8b5a"
format_stamp: "Formatted at 2026-11-19 09:04:27 on dist-test-slave-1fcg"
I20261119 09:04:27.550115 3368 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:27.577606 3368 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:27.578117 3368 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:27.578274 3368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:27.578780 3368 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:27.579818 3368 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261119 09:04:27.579866 3368 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:27.580116 3368 ts_tablet_manager.cc:662] Registered 0 tablets
I20261119 09:04:27.580174 3368 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:27.589851 3368 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.68:38007
I20261119 09:04:27.589947 3495 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.68:38007 every 8 connection(s)
I20261119 09:04:27.590983 3368 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20261119 09:04:27.598986 3496 heartbeater.cc:345] Connected to a master server at 127.30.252.126:35689
I20261119 09:04:27.599262 3496 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:27.599862 3496 heartbeater.cc:508] Master 127.30.252.126:35689 requested a full tablet report, sending...
I20261119 09:04:27.599871 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 3368
I20261119 09:04:27.599980 31729 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20261119 09:04:27.601117 2913 ts_manager.cc:210] Registered new tserver with Master: 1a51220b262b4a93b5e26e30bb3e8b5a (127.30.252.68:38007)
I20261119 09:04:27.601882 2913 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.68:40327
I20261119 09:04:27.602465 31729 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261119 09:04:27.612383 2914 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56226:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20261119 09:04:27.632325 3430 tablet_service.cc:1495] Processing CreateTablet for tablet c5aa30792cc64eca9d971901a2cef5f9 (DEFAULT_TABLE table=TestTable [id=39f958d15933408497f0dddcd635fc17]), partition=RANGE (key) PARTITION UNBOUNDED
I20261119 09:04:27.632325 3036 tablet_service.cc:1495] Processing CreateTablet for tablet c5aa30792cc64eca9d971901a2cef5f9 (DEFAULT_TABLE table=TestTable [id=39f958d15933408497f0dddcd635fc17]), partition=RANGE (key) PARTITION UNBOUNDED
I20261119 09:04:27.632328 3299 tablet_service.cc:1495] Processing CreateTablet for tablet c5aa30792cc64eca9d971901a2cef5f9 (DEFAULT_TABLE table=TestTable [id=39f958d15933408497f0dddcd635fc17]), partition=RANGE (key) PARTITION UNBOUNDED
I20261119 09:04:27.633303 3430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5aa30792cc64eca9d971901a2cef5f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:27.633307 3036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5aa30792cc64eca9d971901a2cef5f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:27.633338 3299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5aa30792cc64eca9d971901a2cef5f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20261119 09:04:27.638831 3513 tablet_bootstrap.cc:492] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a: Bootstrap starting.
I20261119 09:04:27.640026 3514 tablet_bootstrap.cc:492] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8: Bootstrap starting.
I20261119 09:04:27.641819 3513 tablet_bootstrap.cc:654] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:27.642297 3514 tablet_bootstrap.cc:654] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:27.642759 3513 log.cc:826] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:27.643111 3514 log.cc:826] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:27.643487 3515 tablet_bootstrap.cc:492] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e: Bootstrap starting.
I20261119 09:04:27.644390 3514 tablet_bootstrap.cc:492] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8: No bootstrap required, opened a new log
I20261119 09:04:27.644426 3513 tablet_bootstrap.cc:492] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a: No bootstrap required, opened a new log
I20261119 09:04:27.644619 3514 ts_tablet_manager.cc:1449] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20261119 09:04:27.644620 3513 ts_tablet_manager.cc:1449] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20261119 09:04:27.645861 3515 tablet_bootstrap.cc:654] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e: Neither blocks nor log segments found. Creating new log.
I20261119 09:04:27.647061 3515 log.cc:826] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:27.647914 3514 raft_consensus.cc:360] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } }
I20261119 09:04:27.648115 3514 raft_consensus.cc:386] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:04:27.648171 3514 raft_consensus.cc:748] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d774fe05a5d048fd9b410188ad7302b8, State: Initialized, Role: FOLLOWER
I20261119 09:04:27.648063 3513 raft_consensus.cc:360] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } }
I20261119 09:04:27.648267 3513 raft_consensus.cc:386] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:04:27.648336 3513 raft_consensus.cc:748] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a51220b262b4a93b5e26e30bb3e8b5a, State: Initialized, Role: FOLLOWER
I20261119 09:04:27.648659 3514 consensus_queue.cc:261] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } }
I20261119 09:04:27.648732 3515 tablet_bootstrap.cc:492] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e: No bootstrap required, opened a new log
I20261119 09:04:27.648729 3513 consensus_queue.cc:261] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } }
I20261119 09:04:27.648943 3515 ts_tablet_manager.cc:1449] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20261119 09:04:27.649395 3514 ts_tablet_manager.cc:1480] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20261119 09:04:27.649570 3513 ts_tablet_manager.cc:1480] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20261119 09:04:27.649842 3102 heartbeater.cc:500] Master 127.30.252.126:35689 was elected leader, sending a full tablet report...
I20261119 09:04:27.650945 3496 heartbeater.cc:500] Master 127.30.252.126:35689 was elected leader, sending a full tablet report...
I20261119 09:04:27.651701 3515 raft_consensus.cc:360] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } }
I20261119 09:04:27.651871 3515 raft_consensus.cc:386] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261119 09:04:27.652004 3515 raft_consensus.cc:748] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c7f7565a51d4e2f9b80bb3aff51fd8e, State: Initialized, Role: FOLLOWER
I20261119 09:04:27.652456 3515 consensus_queue.cc:261] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } }
I20261119 09:04:27.653107 3365 heartbeater.cc:500] Master 127.30.252.126:35689 was elected leader, sending a full tablet report...
W20261119 09:04:27.653093 3103 tablet.cc:2554] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261119 09:04:27.653285 3515 ts_tablet_manager.cc:1480] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20261119 09:04:27.691820 3497 tablet.cc:2554] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20261119 09:04:27.707924 3366 tablet.cc:2554] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261119 09:04:27.745211 3520 raft_consensus.cc:499] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261119 09:04:27.745463 3520 raft_consensus.cc:521] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } }
I20261119 09:04:27.746879 3520 leader_election.cc:290] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d774fe05a5d048fd9b410188ad7302b8 (127.30.252.65:34525), 8c7f7565a51d4e2f9b80bb3aff51fd8e (127.30.252.67:42671)
I20261119 09:04:27.751066 3056 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "c5aa30792cc64eca9d971901a2cef5f9" candidate_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d774fe05a5d048fd9b410188ad7302b8" is_pre_election: true
I20261119 09:04:27.751397 3056 raft_consensus.cc:2490] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a51220b262b4a93b5e26e30bb3e8b5a in term 0.
I20261119 09:04:27.751749 3319 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "c5aa30792cc64eca9d971901a2cef5f9" candidate_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" is_pre_election: true
I20261119 09:04:27.751922 3386 leader_election.cc:304] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a51220b262b4a93b5e26e30bb3e8b5a, d774fe05a5d048fd9b410188ad7302b8; no voters:
I20261119 09:04:27.751977 3319 raft_consensus.cc:2490] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a51220b262b4a93b5e26e30bb3e8b5a in term 0.
I20261119 09:04:27.752158 3520 raft_consensus.cc:2826] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261119 09:04:27.752221 3520 raft_consensus.cc:499] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261119 09:04:27.752270 3520 raft_consensus.cc:3096] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:27.753201 3520 raft_consensus.cc:521] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } }
I20261119 09:04:27.753556 3520 leader_election.cc:290] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [CANDIDATE]: Term 1 election: Requested vote from peers d774fe05a5d048fd9b410188ad7302b8 (127.30.252.65:34525), 8c7f7565a51d4e2f9b80bb3aff51fd8e (127.30.252.67:42671)
I20261119 09:04:27.753844 3056 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "c5aa30792cc64eca9d971901a2cef5f9" candidate_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d774fe05a5d048fd9b410188ad7302b8"
I20261119 09:04:27.753880 3319 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "c5aa30792cc64eca9d971901a2cef5f9" candidate_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e"
I20261119 09:04:27.753974 3056 raft_consensus.cc:3096] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:27.753989 3319 raft_consensus.cc:3096] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [term 0 FOLLOWER]: Advancing to term 1
I20261119 09:04:27.754794 3056 raft_consensus.cc:2490] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a51220b262b4a93b5e26e30bb3e8b5a in term 1.
I20261119 09:04:27.754807 3319 raft_consensus.cc:2490] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a51220b262b4a93b5e26e30bb3e8b5a in term 1.
I20261119 09:04:27.755069 3383 leader_election.cc:304] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a51220b262b4a93b5e26e30bb3e8b5a, 8c7f7565a51d4e2f9b80bb3aff51fd8e; no voters:
I20261119 09:04:27.755234 3520 raft_consensus.cc:2826] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [term 1 FOLLOWER]: Leader election won for term 1
I20261119 09:04:27.755447 3520 raft_consensus.cc:705] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [term 1 LEADER]: Becoming Leader. State: Replica: 1a51220b262b4a93b5e26e30bb3e8b5a, State: Running, Role: LEADER
I20261119 09:04:27.755781 3520 consensus_queue.cc:238] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } }
I20261119 09:04:27.759078 3520 consensus_queue.cc:1049] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261119 09:04:27.759397 3525 consensus_queue.cc:1049] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261119 09:04:27.760030 2913 catalog_manager.cc:5856] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a51220b262b4a93b5e26e30bb3e8b5a (127.30.252.68). New cstate: current_term: 1 leader_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } health_report { overall_health: HEALTHY } } }
I20261119 09:04:27.779304 31729 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261119 09:04:27.780678 31729 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d774fe05a5d048fd9b410188ad7302b8 to finish bootstrapping
I20261119 09:04:27.786301 31729 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8c7f7565a51d4e2f9b80bb3aff51fd8e to finish bootstrapping
I20261119 09:04:27.791607 31729 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1a51220b262b4a93b5e26e30bb3e8b5a to finish bootstrapping
I20261119 09:04:27.796521 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 3368
I20261119 09:04:27.802289 31729 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
/tmp/dist-test-taskH9Gyab/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.252.68:38007
--local_ip_for_outbound_sockets=127.30.252.68
--tserver_master_addrs=127.30.252.126:35689
--webserver_port=45875
--webserver_interface=127.30.252.68
--builtin_ntp_servers=127.30.252.84:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000
--tablet_bootstrap_inject_latency_ms=3000 with env {}
W20261119 09:04:27.905524 3534 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261119 09:04:27.905764 3534 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261119 09:04:27.905807 3534 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261119 09:04:27.905874 3534 flags.cc:432] Enabled unsafe flag: --tablet_bootstrap_inject_latency_ms=3000
W20261119 09:04:27.905903 3534 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261119 09:04:27.909361 3534 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261119 09:04:27.909415 3534 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261119 09:04:27.909461 3534 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261119 09:04:27.909569 3534 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261119 09:04:27.909610 3534 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.252.68
I20261119 09:04:27.913576 3534 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.252.84:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.252.68:38007
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.30.252.68
--webserver_port=45875
--tserver_master_addrs=127.30.252.126:35689
--tablet_bootstrap_inject_latency_ms=3000
--never_fsync=true
--heap_profile_path=/tmp/kudu.3534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.252.68
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision a786e784452aff5339cfb7c8e1343a261d1f6aff
build type DEBUG
built by None at 19 Nov 2026 08:43:13 UTC on bdcb31816ec0
build id 14855
I20261119 09:04:27.914354 3534 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261119 09:04:27.915035 3534 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261119 09:04:27.920696 3539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:27.920670 3542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261119 09:04:27.920714 3540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261119 09:04:27.921084 3534 server_base.cc:1061] running on GCE node
I20261119 09:04:27.921384 3534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261119 09:04:27.921907 3534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261119 09:04:27.923072 3534 hybrid_clock.cc:648] HybridClock initialized: now 1795079067923057 us; error 28 us; skew 500 ppm
I20261119 09:04:27.925041 3534 webserver.cc:577] Webserver started at http://127.30.252.68:45875/ using document root <none> and password file <none>
I20261119 09:04:27.925738 3534 fs_manager.cc:362] Metadata directory not provided
I20261119 09:04:27.925897 3534 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261119 09:04:27.929400 3534 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20261119 09:04:27.931617 3548 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261119 09:04:27.932597 3534 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261119 09:04:27.932705 3534 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1a51220b262b4a93b5e26e30bb3e8b5a"
format_stamp: "Formatted at 2026-11-19 09:04:27 on dist-test-slave-1fcg"
I20261119 09:04:27.933060 3534 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261119 09:04:27.966351 3534 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261119 09:04:27.966883 3534 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261119 09:04:27.967026 3534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261119 09:04:27.967525 3534 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261119 09:04:27.968837 3555 ts_tablet_manager.cc:588] Loading tablet metadata (0/1 complete)
I20261119 09:04:27.972343 3534 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261119 09:04:27.972417 3534 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20261119 09:04:27.972488 3534 ts_tablet_manager.cc:646] Registering tablets (0/1 complete)
I20261119 09:04:27.973989 3534 ts_tablet_manager.cc:662] Registered 1 tablets
I20261119 09:04:27.974030 3534 ts_tablet_manager.cc:641] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
W20261119 09:04:27.974131 3555 ts_tablet_manager.cc:1428] Injecting 3000ms delay in tablet bootstrapping
I20261119 09:04:27.983240 3534 rpc_server.cc:307] RPC server started. Bound to: 127.30.252.68:38007
I20261119 09:04:27.983273 3662 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.252.68:38007 every 8 connection(s)
I20261119 09:04:27.984248 3534 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20261119 09:04:27.988353 31729 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu as pid 3534
I20261119 09:04:27.993618 3663 heartbeater.cc:345] Connected to a master server at 127.30.252.126:35689
I20261119 09:04:27.993896 3663 heartbeater.cc:462] Registering TS with master...
I20261119 09:04:27.994449 3663 heartbeater.cc:508] Master 127.30.252.126:35689 requested a full tablet report, sending...
I20261119 09:04:27.996237 2912 ts_manager.cc:210] Re-registered known tserver with Master: 1a51220b262b4a93b5e26e30bb3e8b5a (127.30.252.68:38007)
I20261119 09:04:27.997159 2912 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.30.252.68:37889
I20261119 09:04:28.224591 3234 heartbeater.cc:500] Master 127.30.252.126:35689 was elected leader, sending a full tablet report...
I20261119 09:04:28.999317 3663 heartbeater.cc:500] Master 127.30.252.126:35689 was elected leader, sending a full tablet report...
I20261119 09:04:29.264964 3668 raft_consensus.cc:499] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1a51220b262b4a93b5e26e30bb3e8b5a)
I20261119 09:04:29.265120 3668 raft_consensus.cc:521] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } }
I20261119 09:04:29.265268 3669 raft_consensus.cc:499] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1a51220b262b4a93b5e26e30bb3e8b5a)
I20261119 09:04:29.265390 3669 raft_consensus.cc:521] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } }
I20261119 09:04:29.266310 3668 leader_election.cc:290] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c7f7565a51d4e2f9b80bb3aff51fd8e (127.30.252.67:42671), 1a51220b262b4a93b5e26e30bb3e8b5a (127.30.252.68:38007)
I20261119 09:04:29.266320 3669 leader_election.cc:290] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d774fe05a5d048fd9b410188ad7302b8 (127.30.252.65:34525), 1a51220b262b4a93b5e26e30bb3e8b5a (127.30.252.68:38007)
I20261119 09:04:29.270829 3319 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "c5aa30792cc64eca9d971901a2cef5f9" candidate_uuid: "d774fe05a5d048fd9b410188ad7302b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" is_pre_election: true
I20261119 09:04:29.270990 3319 raft_consensus.cc:2490] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d774fe05a5d048fd9b410188ad7302b8 in term 1.
I20261119 09:04:29.271245 3056 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "c5aa30792cc64eca9d971901a2cef5f9" candidate_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d774fe05a5d048fd9b410188ad7302b8" is_pre_election: true
I20261119 09:04:29.271374 3056 raft_consensus.cc:2490] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c7f7565a51d4e2f9b80bb3aff51fd8e in term 1.
I20261119 09:04:29.271402 2989 leader_election.cc:304] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c7f7565a51d4e2f9b80bb3aff51fd8e, d774fe05a5d048fd9b410188ad7302b8; no voters:
I20261119 09:04:29.271765 3255 leader_election.cc:304] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c7f7565a51d4e2f9b80bb3aff51fd8e, d774fe05a5d048fd9b410188ad7302b8; no voters:
I20261119 09:04:29.271804 3668 raft_consensus.cc:2826] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261119 09:04:29.271873 3668 raft_consensus.cc:499] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1a51220b262b4a93b5e26e30bb3e8b5a)
I20261119 09:04:29.271914 3668 raft_consensus.cc:3096] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 1 FOLLOWER]: Advancing to term 2
I20261119 09:04:29.271991 3669 raft_consensus.cc:2826] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261119 09:04:29.272035 3669 raft_consensus.cc:499] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1a51220b262b4a93b5e26e30bb3e8b5a)
I20261119 09:04:29.272069 3669 raft_consensus.cc:3096] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [term 1 FOLLOWER]: Advancing to term 2
I20261119 09:04:29.272758 3668 raft_consensus.cc:521] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } }
I20261119 09:04:29.271787 3617 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "c5aa30792cc64eca9d971901a2cef5f9" candidate_uuid: "d774fe05a5d048fd9b410188ad7302b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" is_pre_election: true
I20261119 09:04:29.272058 3616 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "c5aa30792cc64eca9d971901a2cef5f9" candidate_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" is_pre_election: true
I20261119 09:04:29.273054 3669 raft_consensus.cc:521] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } }
I20261119 09:04:29.273123 3668 leader_election.cc:290] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [CANDIDATE]: Term 2 election: Requested vote from peers 8c7f7565a51d4e2f9b80bb3aff51fd8e (127.30.252.67:42671), 1a51220b262b4a93b5e26e30bb3e8b5a (127.30.252.68:38007)
I20261119 09:04:29.273393 3669 leader_election.cc:290] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [CANDIDATE]: Term 2 election: Requested vote from peers d774fe05a5d048fd9b410188ad7302b8 (127.30.252.65:34525), 1a51220b262b4a93b5e26e30bb3e8b5a (127.30.252.68:38007)
I20261119 09:04:29.273692 3617 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "c5aa30792cc64eca9d971901a2cef5f9" candidate_uuid: "d774fe05a5d048fd9b410188ad7302b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a"
W20261119 09:04:29.273687 2989 leader_election.cc:343] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1a51220b262b4a93b5e26e30bb3e8b5a (127.30.252.68:38007): Illegal state: must be running to vote when last-logged opid is not known
I20261119 09:04:29.273851 3056 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "c5aa30792cc64eca9d971901a2cef5f9" candidate_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d774fe05a5d048fd9b410188ad7302b8"
I20261119 09:04:29.273947 3617 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "c5aa30792cc64eca9d971901a2cef5f9" candidate_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a"
I20261119 09:04:29.274005 3056 raft_consensus.cc:2415] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c7f7565a51d4e2f9b80bb3aff51fd8e in current term 2: Already voted for candidate d774fe05a5d048fd9b410188ad7302b8 in this term.
I20261119 09:04:29.274106 3319 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "c5aa30792cc64eca9d971901a2cef5f9" candidate_uuid: "d774fe05a5d048fd9b410188ad7302b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e"
W20261119 09:04:29.274166 2989 leader_election.cc:343] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 1a51220b262b4a93b5e26e30bb3e8b5a (127.30.252.68:38007): Illegal state: must be running to vote when last-logged opid is not known
I20261119 09:04:29.274256 3319 raft_consensus.cc:2415] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d774fe05a5d048fd9b410188ad7302b8 in current term 2: Already voted for candidate 8c7f7565a51d4e2f9b80bb3aff51fd8e in this term.
W20261119 09:04:29.274302 3252 leader_election.cc:343] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 1a51220b262b4a93b5e26e30bb3e8b5a (127.30.252.68:38007): Illegal state: must be running to vote when last-logged opid is not known
I20261119 09:04:29.274433 3255 leader_election.cc:304] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c7f7565a51d4e2f9b80bb3aff51fd8e; no voters: 1a51220b262b4a93b5e26e30bb3e8b5a, d774fe05a5d048fd9b410188ad7302b8
I20261119 09:04:29.274559 2989 leader_election.cc:304] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d774fe05a5d048fd9b410188ad7302b8; no voters: 1a51220b262b4a93b5e26e30bb3e8b5a, 8c7f7565a51d4e2f9b80bb3aff51fd8e
I20261119 09:04:29.274591 3669 raft_consensus.cc:2771] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20261119 09:04:29.274735 3668 raft_consensus.cc:2771] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20261119 09:04:30.974838 3555 tablet_bootstrap.cc:492] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a: Bootstrap starting.
I20261119 09:04:30.983968 3555 log.cc:826] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a: Log is configured to *not* fsync() on all Append() calls
I20261119 09:04:30.986805 3555 tablet_bootstrap.cc:492] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261119 09:04:30.987174 3555 tablet_bootstrap.cc:492] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a: Bootstrap complete.
I20261119 09:04:30.987654 3555 ts_tablet_manager.cc:1449] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a: Time spent bootstrapping tablet: real 0.013s user 0.013s sys 0.000s
I20261119 09:04:30.989816 3555 raft_consensus.cc:360] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } }
I20261119 09:04:30.990191 3555 raft_consensus.cc:748] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a51220b262b4a93b5e26e30bb3e8b5a, State: Initialized, Role: FOLLOWER
I20261119 09:04:30.990702 3555 consensus_queue.cc:261] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } }
I20261119 09:04:30.991613 3555 ts_tablet_manager.cc:1480] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20261119 09:04:31.257458 3680 raft_consensus.cc:499] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261119 09:04:31.257640 3680 raft_consensus.cc:521] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } }
I20261119 09:04:31.258169 3680 leader_election.cc:290] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c7f7565a51d4e2f9b80bb3aff51fd8e (127.30.252.67:42671), 1a51220b262b4a93b5e26e30bb3e8b5a (127.30.252.68:38007)
I20261119 09:04:31.258767 3319 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "c5aa30792cc64eca9d971901a2cef5f9" candidate_uuid: "d774fe05a5d048fd9b410188ad7302b8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" is_pre_election: true
I20261119 09:04:31.258934 3319 raft_consensus.cc:2490] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d774fe05a5d048fd9b410188ad7302b8 in term 2.
I20261119 09:04:31.258909 3617 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "c5aa30792cc64eca9d971901a2cef5f9" candidate_uuid: "d774fe05a5d048fd9b410188ad7302b8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" is_pre_election: true
I20261119 09:04:31.259236 3617 raft_consensus.cc:2490] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d774fe05a5d048fd9b410188ad7302b8 in term 1.
I20261119 09:04:31.259310 2989 leader_election.cc:304] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c7f7565a51d4e2f9b80bb3aff51fd8e, d774fe05a5d048fd9b410188ad7302b8; no voters:
I20261119 09:04:31.259501 3680 raft_consensus.cc:2826] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261119 09:04:31.259591 3680 raft_consensus.cc:499] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261119 09:04:31.259639 3680 raft_consensus.cc:3096] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 2 FOLLOWER]: Advancing to term 3
I20261119 09:04:31.260733 3680 raft_consensus.cc:521] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } }
I20261119 09:04:31.261089 3680 leader_election.cc:290] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [CANDIDATE]: Term 3 election: Requested vote from peers 8c7f7565a51d4e2f9b80bb3aff51fd8e (127.30.252.67:42671), 1a51220b262b4a93b5e26e30bb3e8b5a (127.30.252.68:38007)
I20261119 09:04:31.261479 3319 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "c5aa30792cc64eca9d971901a2cef5f9" candidate_uuid: "d774fe05a5d048fd9b410188ad7302b8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e"
I20261119 09:04:31.261605 3319 raft_consensus.cc:3096] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [term 2 FOLLOWER]: Advancing to term 3
I20261119 09:04:31.261606 3617 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "c5aa30792cc64eca9d971901a2cef5f9" candidate_uuid: "d774fe05a5d048fd9b410188ad7302b8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a"
I20261119 09:04:31.261751 3617 raft_consensus.cc:3096] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [term 1 FOLLOWER]: Advancing to term 3
I20261119 09:04:31.262678 3319 raft_consensus.cc:2490] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d774fe05a5d048fd9b410188ad7302b8 in term 3.
I20261119 09:04:31.263015 2989 leader_election.cc:304] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c7f7565a51d4e2f9b80bb3aff51fd8e, d774fe05a5d048fd9b410188ad7302b8; no voters:
I20261119 09:04:31.263180 3680 raft_consensus.cc:2826] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 3 FOLLOWER]: Leader election won for term 3
I20261119 09:04:31.263437 3680 raft_consensus.cc:705] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [term 3 LEADER]: Becoming Leader. State: Replica: d774fe05a5d048fd9b410188ad7302b8, State: Running, Role: LEADER
I20261119 09:04:31.263990 3617 raft_consensus.cc:2490] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d774fe05a5d048fd9b410188ad7302b8 in term 3.
I20261119 09:04:31.263939 3680 consensus_queue.cc:238] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } }
I20261119 09:04:31.266848 3319 raft_consensus.cc:1285] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e [term 3 FOLLOWER]: Refusing update from remote peer d774fe05a5d048fd9b410188ad7302b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20261119 09:04:31.267066 3617 raft_consensus.cc:1285] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a [term 3 FOLLOWER]: Refusing update from remote peer d774fe05a5d048fd9b410188ad7302b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20261119 09:04:31.267313 2912 catalog_manager.cc:5856] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 reported cstate change: term changed from 1 to 3, leader changed from 1a51220b262b4a93b5e26e30bb3e8b5a (127.30.252.68) to d774fe05a5d048fd9b410188ad7302b8 (127.30.252.65). New cstate: current_term: 3 leader_uuid: "d774fe05a5d048fd9b410188ad7302b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d774fe05a5d048fd9b410188ad7302b8" member_type: VOTER last_known_addr { host: "127.30.252.65" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 } health_report { overall_health: UNKNOWN } } }
I20261119 09:04:31.267433 3681 consensus_queue.cc:1049] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c7f7565a51d4e2f9b80bb3aff51fd8e" member_type: VOTER last_known_addr { host: "127.30.252.67" port: 42671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261119 09:04:31.267838 3681 consensus_queue.cc:1049] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a51220b262b4a93b5e26e30bb3e8b5a" member_type: VOTER last_known_addr { host: "127.30.252.68" port: 38007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261119 09:04:31.674400 31729 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20261119 09:04:31.674816 31729 raft_config_change-itest.cc:610] Waiting until we've written at least 4MB...
I20261119 09:04:31.772236 3103 maintenance_manager.cc:419] P d774fe05a5d048fd9b410188ad7302b8: Scheduling FlushMRSOp(c5aa30792cc64eca9d971901a2cef5f9): 2761400 bytes log retention, and flush 4038921 bytes memory
I20261119 09:04:31.797595 3664 maintenance_manager.cc:419] P 1a51220b262b4a93b5e26e30bb3e8b5a: Scheduling FlushMRSOp(c5aa30792cc64eca9d971901a2cef5f9): 3944631 bytes log retention, and flush 5079305 bytes memory
I20261119 09:04:31.819685 31729 raft_config_change-itest.cc:616] Waiting for log GC on d774fe05a5d048fd9b410188ad7302b8
I20261119 09:04:31.827024 3366 maintenance_manager.cc:419] P 8c7f7565a51d4e2f9b80bb3aff51fd8e: Scheduling FlushMRSOp(c5aa30792cc64eca9d971901a2cef5f9): 4339184 bytes log retention, and flush 6119689 bytes memory
I20261119 09:04:31.856659 2993 maintenance_manager.cc:643] P d774fe05a5d048fd9b410188ad7302b8: FlushMRSOp(c5aa30792cc64eca9d971901a2cef5f9) complete. Timing: real 0.084s user 0.052s sys 0.003s Metrics: {"bytes_written":135937,"cfile_init":1,"compiler_manager_pool.queue_time_us":243,"dirs.queue_time_us":5171,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":647,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":26,"thread_start_us":282,"threads_started":2}
I20261119 09:04:31.857923 3103 maintenance_manager.cc:419] P d774fe05a5d048fd9b410188ad7302b8: Scheduling LogGCOp(c5aa30792cc64eca9d971901a2cef5f9): free 2761400 bytes of WAL
I20261119 09:04:31.875982 2993 log_reader.cc:385] T c5aa30792cc64eca9d971901a2cef5f9: removed 3 log segments from log reader
I20261119 09:04:31.876133 2993 log.cc:1079] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal/wals/c5aa30792cc64eca9d971901a2cef5f9/wal-000000001 (ops 1-9)
I20261119 09:04:31.876210 2993 log.cc:1079] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal/wals/c5aa30792cc64eca9d971901a2cef5f9/wal-000000002 (ops 10-16)
I20261119 09:04:31.876232 2993 log.cc:1079] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal/wals/c5aa30792cc64eca9d971901a2cef5f9/wal-000000003 (ops 17-23)
I20261119 09:04:31.877283 2993 maintenance_manager.cc:643] P d774fe05a5d048fd9b410188ad7302b8: LogGCOp(c5aa30792cc64eca9d971901a2cef5f9) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20261119 09:04:31.877966 3103 maintenance_manager.cc:419] P d774fe05a5d048fd9b410188ad7302b8: Scheduling FlushMRSOp(c5aa30792cc64eca9d971901a2cef5f9): 1840912 bytes log retention, and flush 1958153 bytes memory
I20261119 09:04:31.883164 31729 raft_config_change-itest.cc:620] Log GC complete on d774fe05a5d048fd9b410188ad7302b8
I20261119 09:04:31.901883 2993 maintenance_manager.cc:643] P d774fe05a5d048fd9b410188ad7302b8: FlushMRSOp(c5aa30792cc64eca9d971901a2cef5f9) complete. Timing: real 0.024s user 0.010s sys 0.010s Metrics: {"bytes_written":135803,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":1916,"drs_written":1,"lbm_read_time_us":143,"lbm_reads_lt_1ms":4,"lbm_write_time_us":678,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":11}
I20261119 09:04:31.903210 3103 maintenance_manager.cc:419] P d774fe05a5d048fd9b410188ad7302b8: Scheduling LogGCOp(c5aa30792cc64eca9d971901a2cef5f9): free 1840912 bytes of WAL
I20261119 09:04:31.903903 2993 log_reader.cc:385] T c5aa30792cc64eca9d971901a2cef5f9: removed 2 log segments from log reader
I20261119 09:04:31.903965 2993 log.cc:1079] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal/wals/c5aa30792cc64eca9d971901a2cef5f9/wal-000000004 (ops 24-30)
I20261119 09:04:31.904006 2993 log.cc:1079] T c5aa30792cc64eca9d971901a2cef5f9 P d774fe05a5d048fd9b410188ad7302b8: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-0/wal/wals/c5aa30792cc64eca9d971901a2cef5f9/wal-000000005 (ops 31-37)
I20261119 09:04:31.904690 2993 maintenance_manager.cc:643] P d774fe05a5d048fd9b410188ad7302b8: LogGCOp(c5aa30792cc64eca9d971901a2cef5f9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20261119 09:04:31.904989 3103 maintenance_manager.cc:419] P d774fe05a5d048fd9b410188ad7302b8: Scheduling UndoDeltaBlockGCOp(c5aa30792cc64eca9d971901a2cef5f9): 691 bytes on disk
I20261119 09:04:31.903378 3553 maintenance_manager.cc:643] P 1a51220b262b4a93b5e26e30bb3e8b5a: FlushMRSOp(c5aa30792cc64eca9d971901a2cef5f9) complete. Timing: real 0.105s user 0.065s sys 0.000s Metrics: {"bytes_written":135937,"cfile_init":1,"compiler_manager_pool.queue_time_us":19495,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":756,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":30,"thread_start_us":256,"threads_started":2}
I20261119 09:04:31.905910 3664 maintenance_manager.cc:419] P 1a51220b262b4a93b5e26e30bb3e8b5a: Scheduling LogGCOp(c5aa30792cc64eca9d971901a2cef5f9): free 3944631 bytes of WAL
I20261119 09:04:31.906203 3553 log_reader.cc:385] T c5aa30792cc64eca9d971901a2cef5f9: removed 5 log segments from log reader
I20261119 09:04:31.906262 3553 log.cc:1079] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal/wals/c5aa30792cc64eca9d971901a2cef5f9/wal-000000001 (ops 1-7)
I20261119 09:04:31.906323 3553 log.cc:1079] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal/wals/c5aa30792cc64eca9d971901a2cef5f9/wal-000000002 (ops 8-13)
I20261119 09:04:31.906355 3553 log.cc:1079] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal/wals/c5aa30792cc64eca9d971901a2cef5f9/wal-000000003 (ops 14-20)
I20261119 09:04:31.906384 3553 log.cc:1079] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal/wals/c5aa30792cc64eca9d971901a2cef5f9/wal-000000004 (ops 21-25)
I20261119 09:04:31.906417 3553 log.cc:1079] T c5aa30792cc64eca9d971901a2cef5f9 P 1a51220b262b4a93b5e26e30bb3e8b5a: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-3/wal/wals/c5aa30792cc64eca9d971901a2cef5f9/wal-000000005 (ops 26-32)
I20261119 09:04:31.907786 3553 maintenance_manager.cc:643] P 1a51220b262b4a93b5e26e30bb3e8b5a: LogGCOp(c5aa30792cc64eca9d971901a2cef5f9) complete. Timing: real 0.002s user 0.000s sys 0.002s Metrics: {}
I20261119 09:04:31.908113 3664 maintenance_manager.cc:419] P 1a51220b262b4a93b5e26e30bb3e8b5a: Scheduling UndoDeltaBlockGCOp(c5aa30792cc64eca9d971901a2cef5f9): 470 bytes on disk
I20261119 09:04:31.909036 2993 maintenance_manager.cc:643] P d774fe05a5d048fd9b410188ad7302b8: UndoDeltaBlockGCOp(c5aa30792cc64eca9d971901a2cef5f9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":130,"lbm_reads_lt_1ms":8}
I20261119 09:04:31.909559 3103 maintenance_manager.cc:419] P d774fe05a5d048fd9b410188ad7302b8: Scheduling CompactRowSetsOp(c5aa30792cc64eca9d971901a2cef5f9): perf score=0.978140
I20261119 09:04:31.918426 3553 maintenance_manager.cc:643] P 1a51220b262b4a93b5e26e30bb3e8b5a: UndoDeltaBlockGCOp(c5aa30792cc64eca9d971901a2cef5f9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20261119 09:04:31.921803 3256 maintenance_manager.cc:643] P 8c7f7565a51d4e2f9b80bb3aff51fd8e: FlushMRSOp(c5aa30792cc64eca9d971901a2cef5f9) complete. Timing: real 0.094s user 0.066s sys 0.000s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":16772,"dirs.queue_time_us":16970,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":821,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":720,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":37,"thread_start_us":242,"threads_started":2}
I20261119 09:04:31.922977 3366 maintenance_manager.cc:419] P 8c7f7565a51d4e2f9b80bb3aff51fd8e: Scheduling LogGCOp(c5aa30792cc64eca9d971901a2cef5f9): free 4339184 bytes of WAL
I20261119 09:04:31.923273 3256 log_reader.cc:385] T c5aa30792cc64eca9d971901a2cef5f9: removed 5 log segments from log reader
I20261119 09:04:31.923337 3256 log.cc:1079] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal/wals/c5aa30792cc64eca9d971901a2cef5f9/wal-000000001 (ops 1-8)
I20261119 09:04:31.923406 3256 log.cc:1079] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal/wals/c5aa30792cc64eca9d971901a2cef5f9/wal-000000002 (ops 9-14)
I20261119 09:04:31.923434 3256 log.cc:1079] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal/wals/c5aa30792cc64eca9d971901a2cef5f9/wal-000000003 (ops 15-21)
I20261119 09:04:31.923460 3256 log.cc:1079] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal/wals/c5aa30792cc64eca9d971901a2cef5f9/wal-000000004 (ops 22-28)
I20261119 09:04:31.923487 3256 log.cc:1079] T c5aa30792cc64eca9d971901a2cef5f9 P 8c7f7565a51d4e2f9b80bb3aff51fd8e: Deleting log segment in path: /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0/raft_consensus-itest-cluster/ts-2/wal/wals/c5aa30792cc64eca9d971901a2cef5f9/wal-000000005 (ops 29-35)
I20261119 09:04:31.924927 3256 maintenance_manager.cc:643] P 8c7f7565a51d4e2f9b80bb3aff51fd8e: LogGCOp(c5aa30792cc64eca9d971901a2cef5f9) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20261119 09:04:31.925273 3366 maintenance_manager.cc:419] P 8c7f7565a51d4e2f9b80bb3aff51fd8e: Scheduling UndoDeltaBlockGCOp(c5aa30792cc64eca9d971901a2cef5f9): 550 bytes on disk
I20261119 09:04:31.926064 3256 maintenance_manager.cc:643] P 8c7f7565a51d4e2f9b80bb3aff51fd8e: UndoDeltaBlockGCOp(c5aa30792cc64eca9d971901a2cef5f9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20261119 09:04:31.971442 2993 maintenance_manager.cc:643] P d774fe05a5d048fd9b410188ad7302b8: CompactRowSetsOp(c5aa30792cc64eca9d971901a2cef5f9) complete. Timing: real 0.062s user 0.055s sys 0.004s Metrics: {"bytes_written":136001,"cfile_cache_hit":10,"cfile_cache_hit_bytes":262394,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264454,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":115,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1498,"drs_written":1,"lbm_read_time_us":2291,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":45,"lbm_write_time_us":765,"lbm_writes_lt_1ms":23,"mutex_wait_us":205,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":37,"thread_start_us":428,"threads_started":5}
I20261119 09:04:31.972205 3103 maintenance_manager.cc:419] P d774fe05a5d048fd9b410188ad7302b8: Scheduling UndoDeltaBlockGCOp(c5aa30792cc64eca9d971901a2cef5f9): 550 bytes on disk
I20261119 09:04:31.972711 2993 maintenance_manager.cc:643] P d774fe05a5d048fd9b410188ad7302b8: UndoDeltaBlockGCOp(c5aa30792cc64eca9d971901a2cef5f9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/raft_config_change-itest.cc:500: Failure
Value of: IsRaftConfigMember(added_replica_uuid, cstate.committed_config())
Actual: false
Expected: true
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:402: Failure
Failed
Timed out waiting for assertion to pass.
I20261119 09:05:02.074957 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 2974
I20261119 09:05:02.084007 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 3105
I20261119 09:05:02.091032 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 3237
I20261119 09:05:02.099519 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 3534
I20261119 09:05:02.107447 31729 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskH9Gyab/build/debug/bin/kudu with pid 2883
2026-11-19T09:05:02Z chronyd exiting
I20261119 09:05:02.127223 31729 test_util.cc:182] -----------------------------------------------
I20261119 09:05:02.127315 31729 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taskH9Gyab/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795079038760673-31729-0
[ FAILED ] SlowTabletBootstrapTest.TestFallBehindSlowBootstrap (35478 ms)
[----------] 3 tests from SlowTabletBootstrapTest (47630 ms total)
[----------] Global test environment tear-down
[==========] 6 tests from 2 test suites ran. (63357 ms total)
[ PASSED ] 5 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SlowTabletBootstrapTest.TestFallBehindSlowBootstrap
1 FAILED TEST